《Python3 标准库》电子书和配套代码 "The Python Standard Library by Example" ebook and source code 一本又厚又全的讲解Python3标准库的书籍（1000多页）适合当作字典参考或者闲暇时慢慢阅读。

Pybot can change the way learners try to learn python programming language in a more interactive way. This chatbot will try to solve or provide answer to almost every python related issues or queries that the user is asking for. We are implementing NLP for improving the efficiency of the chatbot. We will include voice feature for more interactivity to the user. By utilizing NLP, developers can organize and structure knowledge to perform tasks such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation. NLTK has been called “a wonderful tool for teaching and working in, computational linguistics using Python,” and “an amazing library to play with natural language.The main issue with text data is that it is all in text format (strings). However, the Machine learning algorithms need some sort of numerical feature vector in order to perform the task. So before we start with any NLP project we need to pre-process it to make it ideal for working. Converting the entire text into uppercase or lowercase, so that the algorithm does not treat the same words in different cases as different Tokenization is just the term used to describe the process of converting the normal text strings into a list of tokens i.e words that we actually want. Sentence tokenizer can be used to find the list of sentences and Word tokenizer can be used to find the list of words in strings.Removing Noise i.e everything that isn’t in a standard number or letter.Removing Stop words. Sometimes, some extremely common words which would appear to be of little value in helping select documents matching a user need are excluded from the vocabulary entirely. These words are called stop words.Stemming is the process of reducing inflected (or sometimes derived) words to their stem, base or root form — generally a written word form. Example if we were to stem the following words: “Stems”, “Stemming”, “Stemmed”, “and Stemtization”, the result would be a single word “stem”. A slight variant of stemming is lemmatization. The major difference between these is, that, stemming can often create non-existent words, whereas lemmas are actual words. So, your root stem, meaning the word you end up with, is not something you can just look up in a dictionary, but you can look up a lemma. Examples of Lemmatization are that “run” is a base form for words like “running” or “ran” or that the word “better” and “good” are in the same lemma so they are considered the same.

What is JavaScript and what does it do? Before you start learning something new, it’s important to understand exactly what it is and what it does. This is especially useful when it comes to mastering a new programming language. In simple terms, JavaScript is a programming language used to make websites interactive. If you think about the basic makeup of a website, you have HTML, which describes and defines the basic content and structure of the website, then you have CSS, which tells the browser how this HTML content should be displayed—determining things like color and font. With just HTML and CSS, you have a website that looks good but doesn’t actually do much. JavaScript brings the website to life by adding functionality. JavaScript is responsible for elements that the user can interact with, such as drop-down menus, modal windows, and contact forms. It is also used to create things like animations, video players, and interactive maps. Nowadays, JavaScript is an all-purpose programming language—meaning it runs across the entire software stack. The most popular application of JavaScript is on the client side (aka frontend), but since Node.js came on the scene, many people run JavaScript on the server side (aka backend) as well. When used on the client side, JavaScript code is read, interpreted, and executed in the user’s web browser. When used on the server side, it is run on a remote computer. You can learn more about the difference between frontend and backend programming here. JavaScript isn’t only used to create websites. It can also be used to build browser-based games and, with the help of certain frameworks, mobile apps for different operating systems. The creation of new libraries and frameworks is also making it possible to build backend programs with JavaScript, such as web apps and server apps. Is it still worth learning JavaScript in 2021? The world of web development is constantly moving. With so many new tools popping up all the time, it can be extremely difficult to know where you should focus your efforts. As an aspiring developer, you’ll want to make sure that what you’re learning is still relevant in today’s industry. If you’re having doubts about JavaScript, it’s important to know that, since its creation in 1995, JavaScript is pretty much everywhere on the web—and that’s not likely to change any time soon. According to the 2020 StackOverflow developer survey, JavaScript is the most commonly used programming language for the eighth year in a row. It is currently used by 94.5% of all websites and, despite originally being designed as a client-side language, JavaScript has now made its way to the server-side of websites (thanks to Node.js), mobile devices (thanks to React Native and Ionic) and desktop (courtesy of Electron). As long as people are interacting with the web, you can assume that JavaScript is highly relevant—there’s no doubt that this is a language worth knowing! With that in mind, let’s look at some of the key benefits of becoming a JavaScript expert. Why learn JavaScript? The most obvious reason for learning JavaScript is if you have hopes of becoming a web developer. Even if you haven’t got your heart set on a tech career, being proficient in JavaScript will enable you to build websites from scratch—a pretty useful skill to have in today’s job market! If you do want to become a web developer, here are some of the main reasons why you should learn JavaScript: JavaScript experts are versatile JavaScript is an extremely versatile language. Once you’ve mastered it, the possibilities are endless: you can code on the client-side (frontend) using Angular and on the server-side (backend) using Node.js. You can also develop web, mobile, and desktop apps using React, React Native, and Electron, and you can even get involved in machine learning. If you want to become a frontend developer, JavaScript is a prerequisite. However, that’s not the only career path open to you as a JavaScript expert. Mastering this key programming language could see you go on to work in full-stack development, games development, information security software engineering, machine learning, and artificial intelligence—to name just a few! Ultimately, if you want any kind of development or engineering career, proficiency in JavaScript is a must. JavaScript experts are in-demand (and well-paid) JavaScript is the most popular programming language in the world, so it’s no wonder that JavaScript is one of the most sought-after skills in the web development industry today. According to the Devskiller IT Skills and Hiring Report 2020, 72% of companies are looking to hire JavaScript experts. Enter the search term “JavaScript” on job site Indeed and you’ll find over 40,000 jobs requiring this skill (in the US). Run the same search on LinkedIn and the results are in excess of 125,000. At the same time, the global demand for JavaScript seems to outweigh the expertise available on the market. According to this 2018 HackerRank report, 48% of employers worldwide need developers with JavaScript skills, while only 42% of student developers claim to be proficient in JavaScript. And, in their most recent report for 2020, HackerRank once again reports that JavaScript is the most popular language that hiring mangers look for in a web developer candidate. Not only are JavaScript experts in demand—they are also well-paid. In the United States, JavaScript developers earn an average yearly salary of $111,953 per year. We’ve covered this topic in more detail in our JavaScript salary guide, but as you can see, learning JavaScript can really boost your earning potential as a developer. JavaScript is beginner-friendly Compared to many other programming languages, JavaScript offers one of the more beginner-friendly entry points into the world of coding. The great thing about JavaScript is that it comes installed on every modern web browser—there’s no need to set up any kind of development environment, which means you can start coding with JavaScript right away! Another advantage of learning JavaScript as your first programming language is that you get instant feedback; with a minimal amount of JavaScript code, you’ll immediately see visible results. There’s also a huge JavaScript community on sites like Stack Overflow, so you’ll find plenty of support as you learn. Not only is JavaScript beginner-friendly; it will also set you up with some extremely valuable transferable skills. JavaScript supports object-oriented, functional, and imperative styles of programming—skills which can be transferred to any new language you might learn later on, such as Python, Java, or C++. JavaScript provides a crucial introduction to key principles and practices that you’ll take with you throughout your career as a developer. Should you learn plain JavaScript first or can you skip to frameworks and libraries? When deciding whether or not to learn JavaScript, what you’re really asking is whether or not you should learn “vanilla” JavaScript. Vanilla JavaScript just means plain JavaScript without any libraries or frameworks. Let’s explore what this means in more detail now. What is meant by vanilla JavaScript, libraries, and frameworks? If you research the term “vanilla JavaScript”, you might run into some confusion; however, all you need to know is that vanilla JavaScript is used to refer to native, standards-based, non-extended JavaScript. There is no difference between vanilla JavaScript and JavaScript—it’s just there to emphasize the usage of plain JavaScript without the use of libraries and frameworks. So what are libraries and frameworks? JavaScript libraries and frameworks both contain sets of prewritten, ready-to-use JavaScript code—but they’re not the same thing. You can think of a framework as your blueprint for building a website: it gives you a structure to work from, and contains ready-made components and tools that help you to build certain elements much quicker than if you were to code them from scratch. Some popular JavaScript frameworks include Angular, React, Vue, and Node.js. Frameworks also contain libraries. Libraries are smaller than frameworks, and tend to be used for more specific cases. A JavaScript library contains sets of JavaScript code which can be called upon to implement certain functions and features. Let’s imagine you want to code a particular element into your website. You could write, say, ten lines of JavaScript from scratch—or you could take the condensed, ready-made version from your chosen JavaScript library. Some examples of JavaScript libraries include jQuery, Lodash, and Underscore. The easiest way to understand how frameworks and libraries work together is to imagine you are building a house. The framework provides the foundation and the structure, while the library enables you to add in ready-made components (like furniture) rather than building your own from scratch. You can learn more about the relationship between languages and libraries in this post explaining the main differences between JavaScript and jQuery. For now, let’s go back to our original question: How important is it to learn vanilla JavaScript? Should you learn vanilla JavaScript first? When it comes to learning JavaScript, it can be tempting to skip ahead to those time-saving frameworks and libraries we just talked about—and many developers do. However, there are many compelling arguments for learning plain JavaScript first. While JavaScript frameworks may help you get the job done quicker, there’s only so far you can go if you don’t understand the core concepts behind these frameworks. Frontend developer Abhishek Nagekar describes how not learning vanilla JavaScript came back to bite him when he started learning the JavaScript frameworks Node and Express: “As I went to write more and more code in Node and Express, I began to get stuck at even the tiniest problems. Suddenly, I was surrounded with words like callbacks, closures, event loop and prototype. It felt like I got a reintroduction to JavaScript, but this time, it was not a toddler playing in its cradle, it was something of a mysterious monster, challenging me on every other step for not having taken it seriously.” The above Tweet references a long-running joke within the developer community, and although it dates way back to 2015, it’s still highly relevant today. If you want to become a developer who can innovate, not just execute, you need to understand the underlying principles of the web—not just the shortcuts. This means learning vanilla JavaScript before you move on to frameworks. In fact, understanding plain JavaScript will help you later on when it comes to deciding whether to use a framework for a certain project, and if so, which framework to use. Why Study JavaScript? JavaScript is one of the 3 languages all web developers must learn: 1. HTML to define the content of web pages 2. CSS to specify the layout of web pages 3. JavaScript to program the behavior of web pages Learning Speed In this tutorial, the learning speed is your choice. Everything is up to you. If you are struggling, take a break, or re-read the material. Always make sure you understand all the "Try-it-Yourself" examples. The only way to become a clever programmer is to: Practice. Practice. Practice. Code. Code. Code ! Commonly Asked Questions How do I get JavaScript? Where can I download JavaScript? Is JavaScript Free? You don't have to get or download JavaScript. JavaScript is already running in your browser on your computer, on your tablet, and on your smart-phone. JavaScript is free to use for everyone.

Based on Python Module of the Week and The Python Standard Library by Example, I have written out explanations and examples of all(most?) of the standard library modules for Python2.

Quickstart tutorial Prerequisites Before reading this tutorial you should know a bit of Python. If you would like to refresh your memory, take a look at the Python tutorial. If you wish to work the examples in this tutorial, you must also have some software installed on your computer. Please see https://scipy.org/install.html for instructions. Learner profile This tutorial is intended as a quick overview of algebra and arrays in NumPy and want to understand how n-dimensional (n>=2) arrays are represented and can be manipulated. In particular, if you don’t know how to apply common functions to n-dimensional arrays (without using for-loops), or if you want to understand axis and shape properties for n-dimensional arrays, this tutorial might be of help. Learning Objectives After this tutorial, you should be able to: Understand the difference between one-, two- and n-dimensional arrays in NumPy; Understand how to apply some linear algebra operations to n-dimensional arrays without using for-loops; Understand axis and shape properties for n-dimensional arrays. The Basics NumPy’s main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of non-negative integers. In NumPy dimensions are called axes. For example, the coordinates of a point in 3D space [1, 2, 1] has one axis. That axis has 3 elements in it, so we say it has a length of 3. In the example pictured below, the array has 2 axes. The first axis has a length of 2, the second axis has a length of 3. [[ 1., 0., 0.], [ 0., 1., 2.]] NumPy’s array class is called ndarray. It is also known by the alias array. Note that numpy.array is not the same as the Standard Python Library class array.array, which only handles one-dimensional arrays and offers less functionality. The more important attributes of an ndarray object are: ndarray.ndim the number of axes (dimensions) of the array. ndarray.shape the dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the number of axes, ndim. ndarray.size the total number of elements of the array. This is equal to the product of the elements of shape. ndarray.dtype an object describing the type of the elements in the array. One can create or specify dtype’s using standard Python types. Additionally NumPy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples. ndarray.itemsize the size in bytes of each element of the array. For example, an array of elements of type float64 has itemsize 8 (=64/8), while one of type complex32 has itemsize 4 (=32/8). It is equivalent to ndarray.dtype.itemsize. ndarray.data the buffer containing the actual elements of the array. Normally, we won’t need to use this attribute because we will access the elements in an array using indexing facilities. An example >>> import numpy as np a = np.arange(15).reshape(3, 5) a array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14]]) a.shape (3, 5) a.ndim 2 a.dtype.name 'int64' a.itemsize 8 a.size 15 type(a) <class 'numpy.ndarray'> b = np.array([6, 7, 8]) b array([6, 7, 8]) type(b) <class 'numpy.ndarray'> Array Creation There are several ways to create arrays. For example, you can create an array from a regular Python list or tuple using the array function. The type of the resulting array is deduced from the type of the elements in the sequences. >>> >>> import numpy as np >>> a = np.array([2,3,4]) >>> a array([2, 3, 4]) >>> a.dtype dtype('int64') >>> b = np.array([1.2, 3.5, 5.1]) >>> b.dtype dtype('float64') A frequent error consists in calling array with multiple arguments, rather than providing a single sequence as an argument. >>> >>> a = np.array(1,2,3,4) # WRONG Traceback (most recent call last): ... TypeError: array() takes from 1 to 2 positional arguments but 4 were given >>> a = np.array([1,2,3,4]) # RIGHT array transforms sequences of sequences into two-dimensional arrays, sequences of sequences of sequences into three-dimensional arrays, and so on. >>> >>> b = np.array([(1.5,2,3), (4,5,6)]) >>> b array([[1.5, 2. , 3. ], [4. , 5. , 6. ]]) The type of the array can also be explicitly specified at creation time: >>> >>> c = np.array( [ [1,2], [3,4] ], dtype=complex ) >>> c array([[1.+0.j, 2.+0.j], [3.+0.j, 4.+0.j]]) Often, the elements of an array are originally unknown, but its size is known. Hence, NumPy offers several functions to create arrays with initial placeholder content. These minimize the necessity of growing arrays, an expensive operation. The function zeros creates an array full of zeros, the function ones creates an array full of ones, and the function empty creates an array whose initial content is random and depends on the state of the memory. By default, the dtype of the created array is float64. >>> >>> np.zeros((3, 4)) array([[0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]]) >>> np.ones( (2,3,4), dtype=np.int16 ) # dtype can also be specified array([[[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]], [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]], dtype=int16) >>> np.empty( (2,3) ) # uninitialized array([[ 3.73603959e-262, 6.02658058e-154, 6.55490914e-260], # may vary [ 5.30498948e-313, 3.14673309e-307, 1.00000000e+000]]) To create sequences of numbers, NumPy provides the arange function which is analogous to the Python built-in range, but returns an array. >>> >>> np.arange( 10, 30, 5 ) array([10, 15, 20, 25]) >>> np.arange( 0, 2, 0.3 ) # it accepts float arguments array([0. , 0.3, 0.6, 0.9, 1.2, 1.5, 1.8]) When arange is used with floating point arguments, it is generally not possible to predict the number of elements obtained, due to the finite floating point precision. For this reason, it is usually better to use the function linspace that receives as an argument the number of elements that we want, instead of the step: >>> >>> from numpy import pi >>> np.linspace( 0, 2, 9 ) # 9 numbers from 0 to 2 array([0. , 0.25, 0.5 , 0.75, 1. , 1.25, 1.5 , 1.75, 2. ]) >>> x = np.linspace( 0, 2*pi, 100 ) # useful to evaluate function at lots of points >>> f = np.sin(x) See also array, zeros, zeros_like, ones, ones_like, empty, empty_like, arange, linspace, numpy.random.Generator.rand, numpy.random.Generator.randn, fromfunction, fromfile Printing Arrays When you print an array, NumPy displays it in a similar way to nested lists, but with the following layout: the last axis is printed from left to right, the second-to-last is printed from top to bottom, the rest are also printed from top to bottom, with each slice separated from the next by an empty line. One-dimensional arrays are then printed as rows, bidimensionals as matrices and tridimensionals as lists of matrices. >>> >>> a = np.arange(6) # 1d array >>> print(a) [0 1 2 3 4 5] >>> >>> b = np.arange(12).reshape(4,3) # 2d array >>> print(b) [[ 0 1 2] [ 3 4 5] [ 6 7 8] [ 9 10 11]] >>> >>> c = np.arange(24).reshape(2,3,4) # 3d array >>> print(c) [[[ 0 1 2 3] [ 4 5 6 7] [ 8 9 10 11]] [[12 13 14 15] [16 17 18 19] [20 21 22 23]]] See below to get more details on reshape. If an array is too large to be printed, NumPy automatically skips the central part of the array and only prints the corners: >>> >>> print(np.arange(10000)) [ 0 1 2 ... 9997 9998 9999] >>> >>> print(np.arange(10000).reshape(100,100)) [[ 0 1 2 ... 97 98 99] [ 100 101 102 ... 197 198 199] [ 200 201 202 ... 297 298 299] ... [9700 9701 9702 ... 9797 9798 9799] [9800 9801 9802 ... 9897 9898 9899] [9900 9901 9902 ... 9997 9998 9999]] To disable this behaviour and force NumPy to print the entire array, you can change the printing options using set_printoptions. >>> >>> np.set_printoptions(threshold=sys.maxsize) # sys module should be imported Basic Operations Arithmetic operators on arrays apply elementwise. A new array is created and filled with the result. >>> >>> a = np.array( [20,30,40,50] ) >>> b = np.arange( 4 ) >>> b array([0, 1, 2, 3]) >>> c = a-b >>> c array([20, 29, 38, 47]) >>> b**2 array([0, 1, 4, 9]) >>> 10*np.sin(a) array([ 9.12945251, -9.88031624, 7.4511316 , -2.62374854]) >>> a<35 array([ True, True, False, False]) Unlike in many matrix languages, the product operator * operates elementwise in NumPy arrays. The matrix product can be performed using the @ operator (in python >=3.5) or the dot function or method: >>> >>> A = np.array( [[1,1], ... [0,1]] ) >>> B = np.array( [[2,0], ... [3,4]] ) >>> A * B # elementwise product array([[2, 0], [0, 4]]) >>> A @ B # matrix product array([[5, 4], [3, 4]]) >>> A.dot(B) # another matrix product array([[5, 4], [3, 4]]) Some operations, such as += and *=, act in place to modify an existing array rather than create a new one. >>> >>> rg = np.random.default_rng(1) # create instance of default random number generator >>> a = np.ones((2,3), dtype=int) >>> b = rg.random((2,3)) >>> a *= 3 >>> a array([[3, 3, 3], [3, 3, 3]]) >>> b += a >>> b array([[3.51182162, 3.9504637 , 3.14415961], [3.94864945, 3.31183145, 3.42332645]]) >>> a += b # b is not automatically converted to integer type Traceback (most recent call last): ... numpy.core._exceptions.UFuncTypeError: Cannot cast ufunc 'add' output from dtype('float64') to dtype('int64') with casting rule 'same_kind' When operating with arrays of different types, the type of the resulting array corresponds to the more general or precise one (a behavior known as upcasting). >>> >>> a = np.ones(3, dtype=np.int32) >>> b = np.linspace(0,pi,3) >>> b.dtype.name 'float64' >>> c = a+b >>> c array([1. , 2.57079633, 4.14159265]) >>> c.dtype.name 'float64' >>> d = np.exp(c*1j) >>> d array([ 0.54030231+0.84147098j, -0.84147098+0.54030231j, -0.54030231-0.84147098j]) >>> d.dtype.name 'complex128' Many unary operations, such as computing the sum of all the elements in the array, are implemented as methods of the ndarray class. >>> >>> a = rg.random((2,3)) >>> a array([[0.82770259, 0.40919914, 0.54959369], [0.02755911, 0.75351311, 0.53814331]]) >>> a.sum() 3.1057109529998157 >>> a.min() 0.027559113243068367 >>> a.max() 0.8277025938204418 By default, these operations apply to the array as though it were a list of numbers, regardless of its shape. However, by specifying the axis parameter you can apply an operation along the specified axis of an array: >>> >>> b = np.arange(12).reshape(3,4) >>> b array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> >>> b.sum(axis=0) # sum of each column array([12, 15, 18, 21]) >>> >>> b.min(axis=1) # min of each row array([0, 4, 8]) >>> >>> b.cumsum(axis=1) # cumulative sum along each row array([[ 0, 1, 3, 6], [ 4, 9, 15, 22], [ 8, 17, 27, 38]]) Universal Functions NumPy provides familiar mathematical functions such as sin, cos, and exp. In NumPy, these are called “universal functions”(ufunc). Within NumPy, these functions operate elementwise on an array, producing an array as output. >>> >>> B = np.arange(3) >>> B array([0, 1, 2]) >>> np.exp(B) array([1. , 2.71828183, 7.3890561 ]) >>> np.sqrt(B) array([0. , 1. , 1.41421356]) >>> C = np.array([2., -1., 4.]) >>> np.add(B, C) array([2., 0., 6.]) See also all, any, apply_along_axis, argmax, argmin, argsort, average, bincount, ceil, clip, conj, corrcoef, cov, cross, cumprod, cumsum, diff, dot, floor, inner, invert, lexsort, max, maximum, mean, median, min, minimum, nonzero, outer, prod, re, round, sort, std, sum, trace, transpose, var, vdot, vectorize, where Indexing, Slicing and Iterating One-dimensional arrays can be indexed, sliced and iterated over, much like lists and other Python sequences. >>> >>> a = np.arange(10)**3 >>> a array([ 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]) >>> a[2] 8 >>> a[2:5] array([ 8, 27, 64]) # equivalent to a[0:6:2] = 1000; # from start to position 6, exclusive, set every 2nd element to 1000 >>> a[:6:2] = 1000 >>> a array([1000, 1, 1000, 27, 1000, 125, 216, 343, 512, 729]) >>> a[ : :-1] # reversed a array([ 729, 512, 343, 216, 125, 1000, 27, 1000, 1, 1000]) >>> for i in a: ... print(i**(1/3.)) ... 9.999999999999998 1.0 9.999999999999998 3.0 9.999999999999998 4.999999999999999 5.999999999999999 6.999999999999999 7.999999999999999 8.999999999999998 Multidimensional arrays can have one index per axis. These indices are given in a tuple separated by commas: >>> >>> def f(x,y): ... return 10*x+y ... >>> b = np.fromfunction(f,(5,4),dtype=int) >>> b array([[ 0, 1, 2, 3], [10, 11, 12, 13], [20, 21, 22, 23], [30, 31, 32, 33], [40, 41, 42, 43]]) >>> b[2,3] 23 >>> b[0:5, 1] # each row in the second column of b array([ 1, 11, 21, 31, 41]) >>> b[ : ,1] # equivalent to the previous example array([ 1, 11, 21, 31, 41]) >>> b[1:3, : ] # each column in the second and third row of b array([[10, 11, 12, 13], [20, 21, 22, 23]]) When fewer indices are provided than the number of axes, the missing indices are considered complete slices: >>> >>> b[-1] # the last row. Equivalent to b[-1,:] array([40, 41, 42, 43]) The expression within brackets in b[i] is treated as an i followed by as many instances of : as needed to represent the remaining axes. NumPy also allows you to write this using dots as b[i,...]. The dots (...) represent as many colons as needed to produce a complete indexing tuple. For example, if x is an array with 5 axes, then x[1,2,...] is equivalent to x[1,2,:,:,:], x[...,3] to x[:,:,:,:,3] and x[4,...,5,:] to x[4,:,:,5,:]. >>> >>> c = np.array( [[[ 0, 1, 2], # a 3D array (two stacked 2D arrays) ... [ 10, 12, 13]], ... [[100,101,102], ... [110,112,113]]]) >>> c.shape (2, 2, 3) >>> c[1,...] # same as c[1,:,:] or c[1] array([[100, 101, 102], [110, 112, 113]]) >>> c[...,2] # same as c[:,:,2] array([[ 2, 13], [102, 113]]) Iterating over multidimensional arrays is done with respect to the first axis: >>> >>> for row in b: ... print(row) ... [0 1 2 3] [10 11 12 13] [20 21 22 23] [30 31 32 33] [40 41 42 43] However, if one wants to perform an operation on each element in the array, one can use the flat attribute which is an iterator over all the elements of the array: >>> >>> for element in b.flat: ... print(element) ... 0 1 2 3 10 11 12 13 20 21 22 23 30 31 32 33 40 41 42 43 See also Indexing, Indexing (reference), newaxis, ndenumerate, indices Shape Manipulation Changing the shape of an array An array has a shape given by the number of elements along each axis: >>> >>> a = np.floor(10*rg.random((3,4))) >>> a array([[3., 7., 3., 4.], [1., 4., 2., 2.], [7., 2., 4., 9.]]) >>> a.shape (3, 4) The shape of an array can be changed with various commands. Note that the following three commands all return a modified array, but do not change the original array: >>> >>> a.ravel() # returns the array, flattened array([3., 7., 3., 4., 1., 4., 2., 2., 7., 2., 4., 9.]) >>> a.reshape(6,2) # returns the array with a modified shape array([[3., 7.], [3., 4.], [1., 4.], [2., 2.], [7., 2.], [4., 9.]]) >>> a.T # returns the array, transposed array([[3., 1., 7.], [7., 4., 2.], [3., 2., 4.], [4., 2., 9.]]) >>> a.T.shape (4, 3) >>> a.shape (3, 4) The order of the elements in the array resulting from ravel() is normally “C-style”, that is, the rightmost index “changes the fastest”, so the element after a[0,0] is a[0,1]. If the array is reshaped to some other shape, again the array is treated as “C-style”. NumPy normally creates arrays stored in this order, so ravel() will usually not need to copy its argument, but if the array was made by taking slices of another array or created with unusual options, it may need to be copied. The functions ravel() and reshape() can also be instructed, using an optional argument, to use FORTRAN-style arrays, in which the leftmost index changes the fastest. The reshape function returns its argument with a modified shape, whereas the ndarray.resize method modifies the array itself: >>> >>> a array([[3., 7., 3., 4.], [1., 4., 2., 2.], [7., 2., 4., 9.]]) >>> a.resize((2,6)) >>> a array([[3., 7., 3., 4., 1., 4.], [2., 2., 7., 2., 4., 9.]]) If a dimension is given as -1 in a reshaping operation, the other dimensions are automatically calculated: >>> >>> a.reshape(3,-1) array([[3., 7., 3., 4.], [1., 4., 2., 2.], [7., 2., 4., 9.]]) See also ndarray.shape, reshape, resize, ravel Stacking together different arrays Several arrays can be stacked together along different axes: >>> >>> a = np.floor(10*rg.random((2,2))) >>> a array([[9., 7.], [5., 2.]]) >>> b = np.floor(10*rg.random((2,2))) >>> b array([[1., 9.], [5., 1.]]) >>> np.vstack((a,b)) array([[9., 7.], [5., 2.], [1., 9.], [5., 1.]]) >>> np.hstack((a,b)) array([[9., 7., 1., 9.], [5., 2., 5., 1.]]) The function column_stack stacks 1D arrays as columns into a 2D array. It is equivalent to hstack only for 2D arrays: >>> >>> from numpy import newaxis >>> np.column_stack((a,b)) # with 2D arrays array([[9., 7., 1., 9.], [5., 2., 5., 1.]]) >>> a = np.array([4.,2.]) >>> b = np.array([3.,8.]) >>> np.column_stack((a,b)) # returns a 2D array array([[4., 3.], [2., 8.]]) >>> np.hstack((a,b)) # the result is different array([4., 2., 3., 8.]) >>> a[:,newaxis] # view `a` as a 2D column vector array([[4.], [2.]]) >>> np.column_stack((a[:,newaxis],b[:,newaxis])) array([[4., 3.], [2., 8.]]) >>> np.hstack((a[:,newaxis],b[:,newaxis])) # the result is the same array([[4., 3.], [2., 8.]]) On the other hand, the function row_stack is equivalent to vstack for any input arrays. In fact, row_stack is an alias for vstack: >>> >>> np.column_stack is np.hstack False >>> np.row_stack is np.vstack True In general, for arrays with more than two dimensions, hstack stacks along their second axes, vstack stacks along their first axes, and concatenate allows for an optional arguments giving the number of the axis along which the concatenation should happen. Note In complex cases, r_ and c_ are useful for creating arrays by stacking numbers along one axis. They allow the use of range literals (“:”) >>> >>> np.r_[1:4,0,4] array([1, 2, 3, 0, 4]) When used with arrays as arguments, r_ and c_ are similar to vstack and hstack in their default behavior, but allow for an optional argument giving the number of the axis along which to concatenate. See also hstack, vstack, column_stack, concatenate, c_, r_ Splitting one array into several smaller ones Using hsplit, you can split an array along its horizontal axis, either by specifying the number of equally shaped arrays to return, or by specifying the columns after which the division should occur: >>> >>> a = np.floor(10*rg.random((2,12))) >>> a array([[6., 7., 6., 9., 0., 5., 4., 0., 6., 8., 5., 2.], [8., 5., 5., 7., 1., 8., 6., 7., 1., 8., 1., 0.]]) # Split a into 3 >>> np.hsplit(a,3) [array([[6., 7., 6., 9.], [8., 5., 5., 7.]]), array([[0., 5., 4., 0.], [1., 8., 6., 7.]]), array([[6., 8., 5., 2.], [1., 8., 1., 0.]])] # Split a after the third and the fourth column >>> np.hsplit(a,(3,4)) [array([[6., 7., 6.], [8., 5., 5.]]), array([[9.], [7.]]), array([[0., 5., 4., 0., 6., 8., 5., 2.], [1., 8., 6., 7., 1., 8., 1., 0.]])] vsplit splits along the vertical axis, and array_split allows one to specify along which axis to split. Copies and Views When operating and manipulating arrays, their data is sometimes copied into a new array and sometimes not. This is often a source of confusion for beginners. There are three cases: No Copy at All Simple assignments make no copy of objects or their data. >>> >>> a = np.array([[ 0, 1, 2, 3], ... [ 4, 5, 6, 7], ... [ 8, 9, 10, 11]]) >>> b = a # no new object is created >>> b is a # a and b are two names for the same ndarray object True Python passes mutable objects as references, so function calls make no copy. >>> >>> def f(x): ... print(id(x)) ... >>> id(a) # id is a unique identifier of an object 148293216 # may vary >>> f(a) 148293216 # may vary View or Shallow Copy Different array objects can share the same data. The view method creates a new array object that looks at the same data. >>> >>> c = a.view() >>> c is a False >>> c.base is a # c is a view of the data owned by a True >>> c.flags.owndata False >>> >>> c = c.reshape((2, 6)) # a's shape doesn't change >>> a.shape (3, 4) >>> c[0, 4] = 1234 # a's data changes >>> a array([[ 0, 1, 2, 3], [1234, 5, 6, 7], [ 8, 9, 10, 11]]) Slicing an array returns a view of it: >>> >>> s = a[ : , 1:3] # spaces added for clarity; could also be written "s = a[:, 1:3]" >>> s[:] = 10 # s[:] is a view of s. Note the difference between s = 10 and s[:] = 10 >>> a array([[ 0, 10, 10, 3], [1234, 10, 10, 7], [ 8, 10, 10, 11]]) Deep Copy The copy method makes a complete copy of the array and its data. >>> >>> d = a.copy() # a new array object with new data is created >>> d is a False >>> d.base is a # d doesn't share anything with a False >>> d[0,0] = 9999 >>> a array([[ 0, 10, 10, 3], [1234, 10, 10, 7], [ 8, 10, 10, 11]]) Sometimes copy should be called after slicing if the original array is not required anymore. For example, suppose a is a huge intermediate result and the final result b only contains a small fraction of a, a deep copy should be made when constructing b with slicing: >>> >>> a = np.arange(int(1e8)) >>> b = a[:100].copy() >>> del a # the memory of ``a`` can be released. If b = a[:100] is used instead, a is referenced by b and will persist in memory even if del a is executed. Functions and Methods Overview Here is a list of some useful NumPy functions and methods names ordered in categories. See Routines for the full list. Array Creation arange, array, copy, empty, empty_like, eye, fromfile, fromfunction, identity, linspace, logspace, mgrid, ogrid, ones, ones_like, r_, zeros, zeros_like Conversions ndarray.astype, atleast_1d, atleast_2d, atleast_3d, mat Manipulations array_split, column_stack, concatenate, diagonal, dsplit, dstack, hsplit, hstack, ndarray.item, newaxis, ravel, repeat, reshape, resize, squeeze, swapaxes, take, transpose, vsplit, vstack Questions all, any, nonzero, where Ordering argmax, argmin, argsort, max, min, ptp, searchsorted, sort Operations choose, compress, cumprod, cumsum, inner, ndarray.fill, imag, prod, put, putmask, real, sum Basic Statistics cov, mean, std, var Basic Linear Algebra cross, dot, outer, linalg.svd, vdot Less Basic Broadcasting rules Broadcasting allows universal functions to deal in a meaningful way with inputs that do not have exactly the same shape. The first rule of broadcasting is that if all input arrays do not have the same number of dimensions, a “1” will be repeatedly prepended to the shapes of the smaller arrays until all the arrays have the same number of dimensions. The second rule of broadcasting ensures that arrays with a size of 1 along a particular dimension act as if they had the size of the array with the largest shape along that dimension. The value of the array element is assumed to be the same along that dimension for the “broadcast” array. After application of the broadcasting rules, the sizes of all arrays must match. More details can be found in Broadcasting. Advanced indexing and index tricks NumPy offers more indexing facilities than regular Python sequences. In addition to indexing by integers and slices, as we saw before, arrays can be indexed by arrays of integers and arrays of booleans. Indexing with Arrays of Indices >>> >>> a = np.arange(12)**2 # the first 12 square numbers >>> i = np.array([1, 1, 3, 8, 5]) # an array of indices >>> a[i] # the elements of a at the positions i array([ 1, 1, 9, 64, 25]) >>> >>> j = np.array([[3, 4], [9, 7]]) # a bidimensional array of indices >>> a[j] # the same shape as j array([[ 9, 16], [81, 49]]) When the indexed array a is multidimensional, a single array of indices refers to the first dimension of a. The following example shows this behavior by converting an image of labels into a color image using a palette. >>> >>> palette = np.array([[0, 0, 0], # black ... [255, 0, 0], # red ... [0, 255, 0], # green ... [0, 0, 255], # blue ... [255, 255, 255]]) # white >>> image = np.array([[0, 1, 2, 0], # each value corresponds to a color in the palette ... [0, 3, 4, 0]]) >>> palette[image] # the (2, 4, 3) color image array([[[ 0, 0, 0], [255, 0, 0], [ 0, 255, 0], [ 0, 0, 0]], [[ 0, 0, 0], [ 0, 0, 255], [255, 255, 255], [ 0, 0, 0]]]) We can also give indexes for more than one dimension. The arrays of indices for each dimension must have the same shape. >>> >>> a = np.arange(12).reshape(3,4) >>> a array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> i = np.array([[0, 1], # indices for the first dim of a ... [1, 2]]) >>> j = np.array([[2, 1], # indices for the second dim ... [3, 3]]) >>> >>> a[i, j] # i and j must have equal shape array([[ 2, 5], [ 7, 11]]) >>> >>> a[i, 2] array([[ 2, 6], [ 6, 10]]) >>> >>> a[:, j] # i.e., a[ : , j] array([[[ 2, 1], [ 3, 3]], [[ 6, 5], [ 7, 7]], [[10, 9], [11, 11]]]) In Python, arr[i, j] is exactly the same as arr[(i, j)]—so we can put i and j in a tuple and then do the indexing with that. >>> >>> l = (i, j) # equivalent to a[i, j] >>> a[l] array([[ 2, 5], [ 7, 11]]) However, we can not do this by putting i and j into an array, because this array will be interpreted as indexing the first dimension of a. >>> >>> s = np.array([i, j]) # not what we want >>> a[s] Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: index 3 is out of bounds for axis 0 with size 3 # same as a[i, j] >>> a[tuple(s)] array([[ 2, 5], [ 7, 11]]) Another common use of indexing with arrays is the search of the maximum value of time-dependent series: >>> >>> time = np.linspace(20, 145, 5) # time scale >>> data = np.sin(np.arange(20)).reshape(5,4) # 4 time-dependent series >>> time array([ 20. , 51.25, 82.5 , 113.75, 145. ]) >>> data array([[ 0. , 0.84147098, 0.90929743, 0.14112001], [-0.7568025 , -0.95892427, -0.2794155 , 0.6569866 ], [ 0.98935825, 0.41211849, -0.54402111, -0.99999021], [-0.53657292, 0.42016704, 0.99060736, 0.65028784], [-0.28790332, -0.96139749, -0.75098725, 0.14987721]]) # index of the maxima for each series >>> ind = data.argmax(axis=0) >>> ind array([2, 0, 3, 1]) # times corresponding to the maxima >>> time_max = time[ind] >>> >>> data_max = data[ind, range(data.shape[1])] # => data[ind[0],0], data[ind[1],1]... >>> time_max array([ 82.5 , 20. , 113.75, 51.25]) >>> data_max array([0.98935825, 0.84147098, 0.99060736, 0.6569866 ]) >>> np.all(data_max == data.max(axis=0)) True You can also use indexing with arrays as a target to assign to: >>> >>> a = np.arange(5) >>> a array([0, 1, 2, 3, 4]) >>> a[[1,3,4]] = 0 >>> a array([0, 0, 2, 0, 0]) However, when the list of indices contains repetitions, the assignment is done several times, leaving behind the last value: >>> >>> a = np.arange(5) >>> a[[0,0,2]]=[1,2,3] >>> a array([2, 1, 3, 3, 4]) This is reasonable enough, but watch out if you want to use Python’s += construct, as it may not do what you expect: >>> >>> a = np.arange(5) >>> a[[0,0,2]]+=1 >>> a array([1, 1, 3, 3, 4]) Even though 0 occurs twice in the list of indices, the 0th element is only incremented once. This is because Python requires “a+=1” to be equivalent to “a = a + 1”. Indexing with Boolean Arrays When we index arrays with arrays of (integer) indices we are providing the list of indices to pick. With boolean indices the approach is different; we explicitly choose which items in the array we want and which ones we don’t. The most natural way one can think of for boolean indexing is to use boolean arrays that have the same shape as the original array: >>> >>> a = np.arange(12).reshape(3,4) >>> b = a > 4 >>> b # b is a boolean with a's shape array([[False, False, False, False], [False, True, True, True], [ True, True, True, True]]) >>> a[b] # 1d array with the selected elements array([ 5, 6, 7, 8, 9, 10, 11]) This property can be very useful in assignments: >>> >>> a[b] = 0 # All elements of 'a' higher than 4 become 0 >>> a array([[0, 1, 2, 3], [4, 0, 0, 0], [0, 0, 0, 0]]) You can look at the following example to see how to use boolean indexing to generate an image of the Mandelbrot set: >>> import numpy as np import matplotlib.pyplot as plt def mandelbrot( h,w, maxit=20 ): """Returns an image of the Mandelbrot fractal of size (h,w).""" y,x = np.ogrid[ -1.4:1.4:h*1j, -2:0.8:w*1j ] c = x+y*1j z = c divtime = maxit + np.zeros(z.shape, dtype=int) for i in range(maxit): z = z**2 + c diverge = z*np.conj(z) > 2**2 # who is diverging div_now = diverge & (divtime==maxit) # who is diverging now divtime[div_now] = i # note when z[diverge] = 2 # avoid diverging too much return divtime plt.imshow(mandelbrot(400,400)) ../_images/quickstart-1.png The second way of indexing with booleans is more similar to integer indexing; for each dimension of the array we give a 1D boolean array selecting the slices we want: >>> >>> a = np.arange(12).reshape(3,4) >>> b1 = np.array([False,True,True]) # first dim selection >>> b2 = np.array([True,False,True,False]) # second dim selection >>> >>> a[b1,:] # selecting rows array([[ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> >>> a[b1] # same thing array([[ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> >>> a[:,b2] # selecting columns array([[ 0, 2], [ 4, 6], [ 8, 10]]) >>> >>> a[b1,b2] # a weird thing to do array([ 4, 10]) Note that the length of the 1D boolean array must coincide with the length of the dimension (or axis) you want to slice. In the previous example, b1 has length 3 (the number of rows in a), and b2 (of length 4) is suitable to index the 2nd axis (columns) of a. The ix_() function The ix_ function can be used to combine different vectors so as to obtain the result for each n-uplet. For example, if you want to compute all the a+b*c for all the triplets taken from each of the vectors a, b and c: >>> >>> a = np.array([2,3,4,5]) >>> b = np.array([8,5,4]) >>> c = np.array([5,4,6,8,3]) >>> ax,bx,cx = np.ix_(a,b,c) >>> ax array([[[2]], [[3]], [[4]], [[5]]]) >>> bx array([[[8], [5], [4]]]) >>> cx array([[[5, 4, 6, 8, 3]]]) >>> ax.shape, bx.shape, cx.shape ((4, 1, 1), (1, 3, 1), (1, 1, 5)) >>> result = ax+bx*cx >>> result array([[[42, 34, 50, 66, 26], [27, 22, 32, 42, 17], [22, 18, 26, 34, 14]], [[43, 35, 51, 67, 27], [28, 23, 33, 43, 18], [23, 19, 27, 35, 15]], [[44, 36, 52, 68, 28], [29, 24, 34, 44, 19], [24, 20, 28, 36, 16]], [[45, 37, 53, 69, 29], [30, 25, 35, 45, 20], [25, 21, 29, 37, 17]]]) >>> result[3,2,4] 17 >>> a[3]+b[2]*c[4] 17 You could also implement the reduce as follows: >>> >>> def ufunc_reduce(ufct, *vectors): ... vs = np.ix_(*vectors) ... r = ufct.identity ... for v in vs: ... r = ufct(r,v) ... return r and then use it as: >>> >>> ufunc_reduce(np.add,a,b,c) array([[[15, 14, 16, 18, 13], [12, 11, 13, 15, 10], [11, 10, 12, 14, 9]], [[16, 15, 17, 19, 14], [13, 12, 14, 16, 11], [12, 11, 13, 15, 10]], [[17, 16, 18, 20, 15], [14, 13, 15, 17, 12], [13, 12, 14, 16, 11]], [[18, 17, 19, 21, 16], [15, 14, 16, 18, 13], [14, 13, 15, 17, 12]]]) The advantage of this version of reduce compared to the normal ufunc.reduce is that it makes use of the Broadcasting Rules in order to avoid creating an argument array the size of the output times the number of vectors. Indexing with strings See Structured arrays. Linear Algebra Work in progress. Basic linear algebra to be included here. Simple Array Operations See linalg.py in numpy folder for more. >>> >>> import numpy as np >>> a = np.array([[1.0, 2.0], [3.0, 4.0]]) >>> print(a) [[1. 2.] [3. 4.]] >>> a.transpose() array([[1., 3.], [2., 4.]]) >>> np.linalg.inv(a) array([[-2. , 1. ], [ 1.5, -0.5]]) >>> u = np.eye(2) # unit 2x2 matrix; "eye" represents "I" >>> u array([[1., 0.], [0., 1.]]) >>> j = np.array([[0.0, -1.0], [1.0, 0.0]]) >>> j @ j # matrix product array([[-1., 0.], [ 0., -1.]]) >>> np.trace(u) # trace 2.0 >>> y = np.array([[5.], [7.]]) >>> np.linalg.solve(a, y) array([[-3.], [ 4.]]) >>> np.linalg.eig(j) (array([0.+1.j, 0.-1.j]), array([[0.70710678+0.j , 0.70710678-0.j ], [0. -0.70710678j, 0. +0.70710678j]])) Parameters: square matrix Returns The eigenvalues, each repeated according to its multiplicity. The normalized (unit "length") eigenvectors, such that the column ``v[:,i]`` is the eigenvector corresponding to the eigenvalue ``w[i]`` . Tricks and Tips Here we give a list of short and useful tips. “Automatic” Reshaping To change the dimensions of an array, you can omit one of the sizes which will then be deduced automatically: >>> >>> a = np.arange(30) >>> b = a.reshape((2, -1, 3)) # -1 means "whatever is needed" >>> b.shape (2, 5, 3) >>> b array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11], [12, 13, 14]], [[15, 16, 17], [18, 19, 20], [21, 22, 23], [24, 25, 26], [27, 28, 29]]]) Vector Stacking How do we construct a 2D array from a list of equally-sized row vectors? In MATLAB this is quite easy: if x and y are two vectors of the same length you only need do m=[x;y]. In NumPy this works via the functions column_stack, dstack, hstack and vstack, depending on the dimension in which the stacking is to be done. For example: >>> >>> x = np.arange(0,10,2) >>> y = np.arange(5) >>> m = np.vstack([x,y]) >>> m array([[0, 2, 4, 6, 8], [0, 1, 2, 3, 4]]) >>> xy = np.hstack([x,y]) >>> xy array([0, 2, 4, 6, 8, 0, 1, 2, 3, 4]) The logic behind those functions in more than two dimensions can be strange. See also NumPy for Matlab users Histograms The NumPy histogram function applied to an array returns a pair of vectors: the histogram of the array and a vector of the bin edges. Beware: matplotlib also has a function to build histograms (called hist, as in Matlab) that differs from the one in NumPy. The main difference is that pylab.hist plots the histogram automatically, while numpy.histogram only generates the data. >>> import numpy as np rg = np.random.default_rng(1) import matplotlib.pyplot as plt # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 mu, sigma = 2, 0.5 v = rg.normal(mu,sigma,10000) # Plot a normalized histogram with 50 bins plt.hist(v, bins=50, density=1) # matplotlib version (plot) # Compute the histogram with numpy and then plot it (n, bins) = np.histogram(v, bins=50, density=True) # NumPy version (no plot) plt.plot(.5*(bins[1:]+bins[:-1]), n) ../_images/quickstart-2.png Further reading The Python tutorial NumPy Reference SciPy Tutorial SciPy Lecture Notes A matlab, R, IDL, NumPy/SciPy dictionary © Copyright 2008-2020, The SciPy community. Last updated on Jun 29, 2020. Created using Sphinx 2.4.4.

Dependencies OpenCV The Open Computer Vision library, in the form of python, was used to perform image parsing and analysis. This was used to apply a masking filter to each image, then parse each image for the contours of the marker / target. Numpy The Numerical Python (Numpy) library was used to provide advanced array and matrix functionality for the mathematics used in the program. In particular, the arrays were used for creating and passing the necessary RGB (in this instance, BGR) vectors for creating the appropriate mask. Access The Python program has been designed to run at the allocated hours of each member of Team 4 automatically (See below for a time table). It first starts by retrieving the system time of the executing host and compares it to the hard-coded times for each member. If the system time does not coincide with members' time, the program will inform the user through Standard Output and close. Otherwise, the program will continue with the Execution Procedure. Function Definitions readResponse() Takes the encoded response from the server after issuing a command and parses through the encoded white-space characters for more a more user-friendly report. getCoords(output) Given the output from the server, it parses through the data to find the X, Y, and Z co-ordinates. sendAJMA() Sends the Angular commands to the server to move the arm. After issuing, it acquires the resulting X, Y, and Z co-ordinates from the parsed return output. sendCapture() Sends the capture command to the server and prints the response. find_marker(image) Given an input image, the function parses over each pixel, applying a mask. After massaging the data, it attempts to find contours within the image to find a target marker (for example, a piece of paper). download_image(url, filename) Issues a HTTP request for an image. rawCommand(givenData) Parses through the encoded response from the server and makes it more user-friendly. printData() Does the actual printing of parsed data for the user's benefit. downloadCheckImage() First, it attempts to download the latest image from the server and reports any errors it encounters. Then, it parses the image for the marker if possible and displays the image to the user. testCom() With an initial list of commands for the arm, it proceeds to iterate through them while checking each image for the marker/box. After locating it, it proceeds to center the camera horizontally on the box and then vertically. After having centered the camera on the box, it will proceed to use forward kinematics to calculate the box's location. Execution Procedure The program proceeds to first authenticate with the R12 server. On success, it proceeds to call on the primary driver, testComm, to find the box. This process then instructs the arm for move, capture an image, downloads the image, and parses it for the target. All the while, reporting all pertinent data to the user for diagnostics purposes.

software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. '''A library that provides a Python interface to the Twitter API''' import base64 import calendar import datetime import httplib import os import rfc822 import sys import tempfile import textwrap import time import calendar import urllib import urllib2 import urlparse import gzip import StringIO try: # Python >= 2.6 import json as simplejson except ImportError: try: # Python < 2.6 import simplejson except ImportError: try: # Google App Engine from django.utils import simplejson except ImportError: raise ImportError, "Unable to load a json library" # parse_qsl moved to urlparse module in v2.6 try: from urlparse import parse_qsl, parse_qs except ImportError: from cgi import parse_qsl, parse_qs try: from hashlib import md5 except ImportError: from md5 import md5 import oauth2 as oauth CHARACTER_LIMIT = 140 # A singleton representing a lazily instantiated FileCache. DEFAULT_CACHE = object() REQUEST_TOKEN_URL = 'https://api.twitter.com/oauth/request_token' ACCESS_TOKEN_URL = 'https://api.twitter.com/oauth/access_token' AUTHORIZATION_URL = 'https://api.twitter.com/oauth/authorize' SIGNIN_URL = 'https://api.twitter.com/oauth/authenticate' class TwitterError(Exception): '''Base class for Twitter errors''' @property def message(self): '''Returns the first argument used to construct this error.''' return self.args[0] class Status(object): '''A class representing the Status structure used by the twitter API. The Status structure exposes the following properties: status.created_at status.created_at_in_seconds # read only status.favorited status.in_reply_to_screen_name status.in_reply_to_user_id status.in_reply_to_status_id status.truncated status.source status.id status.text status.location status.relative_created_at # read only status.user status.urls status.user_mentions status.hashtags status.geo status.place status.coordinates status.contributors ''' def __init__(self, created_at=None, favorited=None, id=None, text=None, location=None, user=None, in_reply_to_screen_name=None, in_reply_to_user_id=None, in_reply_to_status_id=None, truncated=None, source=None, now=None, urls=None, user_mentions=None, hashtags=None, geo=None, place=None, coordinates=None, contributors=None, retweeted=None, retweeted_status=None, retweet_count=None): '''An object to hold a Twitter status message. This class is normally instantiated by the twitter.Api class and returned in a sequence. Note: Dates are posted in the form "Sat Jan 27 04:17:38 +0000 2007" Args: created_at: The time this status message was posted. [Optional] favorited: Whether this is a favorite of the authenticated user. [Optional] id: The unique id of this status message. [Optional] text: The text of this status message. [Optional] location: the geolocation string associated with this message. [Optional] relative_created_at: A human readable string representing the posting time. [Optional] user: A twitter.User instance representing the person posting the message. [Optional] now: The current time, if the client choses to set it. Defaults to the wall clock time. [Optional] urls: user_mentions: hashtags: geo: place: coordinates: contributors: retweeted: retweeted_status: retweet_count: ''' self.created_at = created_at self.favorited = favorited self.id = id self.text = text self.location = location self.user = user self.now = now self.in_reply_to_screen_name = in_reply_to_screen_name self.in_reply_to_user_id = in_reply_to_user_id self.in_reply_to_status_id = in_reply_to_status_id self.truncated = truncated self.retweeted = retweeted self.source = source self.urls = urls self.user_mentions = user_mentions self.hashtags = hashtags self.geo = geo self.place = place self.coordinates = coordinates self.contributors = contributors self.retweeted_status = retweeted_status self.retweet_count = retweet_count def GetCreatedAt(self): '''Get the time this status message was posted. Returns: The time this status message was posted ''' return self._created_at def SetCreatedAt(self, created_at): '''Set the time this status message was posted. Args: created_at: The time this status message was created ''' self._created_at = created_at created_at = property(GetCreatedAt, SetCreatedAt, doc='The time this status message was posted.') def GetCreatedAtInSeconds(self): '''Get the time this status message was posted, in seconds since the epoch. Returns: The time this status message was posted, in seconds since the epoch. ''' return calendar.timegm(rfc822.parsedate(self.created_at)) created_at_in_seconds = property(GetCreatedAtInSeconds, doc="The time this status message was " "posted, in seconds since the epoch") def GetFavorited(self): '''Get the favorited setting of this status message. Returns: True if this status message is favorited; False otherwise ''' return self._favorited def SetFavorited(self, favorited): '''Set the favorited state of this status message. Args: favorited: boolean True/False favorited state of this status message ''' self._favorited = favorited favorited = property(GetFavorited, SetFavorited, doc='The favorited state of this status message.') def GetId(self): '''Get the unique id of this status message. Returns: The unique id of this status message ''' return self._id def SetId(self, id): '''Set the unique id of this status message. Args: id: The unique id of this status message ''' self._id = id id = property(GetId, SetId, doc='The unique id of this status message.') def GetInReplyToScreenName(self): return self._in_reply_to_screen_name def SetInReplyToScreenName(self, in_reply_to_screen_name): self._in_reply_to_screen_name = in_reply_to_screen_name in_reply_to_screen_name = property(GetInReplyToScreenName, SetInReplyToScreenName, doc='') def GetInReplyToUserId(self): return self._in_reply_to_user_id def SetInReplyToUserId(self, in_reply_to_user_id): self._in_reply_to_user_id = in_reply_to_user_id in_reply_to_user_id = property(GetInReplyToUserId, SetInReplyToUserId, doc='') def GetInReplyToStatusId(self): return self._in_reply_to_status_id def SetInReplyToStatusId(self, in_reply_to_status_id): self._in_reply_to_status_id = in_reply_to_status_id in_reply_to_status_id = property(GetInReplyToStatusId, SetInReplyToStatusId, doc='') def GetTruncated(self): return self._truncated def SetTruncated(self, truncated): self._truncated = truncated truncated = property(GetTruncated, SetTruncated, doc='') def GetRetweeted(self): return self._retweeted def SetRetweeted(self, retweeted): self._retweeted = retweeted retweeted = property(GetRetweeted, SetRetweeted, doc='') def GetSource(self): return self._source def SetSource(self, source): self._source = source source = property(GetSource, SetSource, doc='') def GetText(self): '''Get the text of this status message. Returns: The text of this status message. ''' return self._text def SetText(self, text): '''Set the text of this status message. Args: text: The text of this status message ''' self._text = text text = property(GetText, SetText, doc='The text of this status message') def GetLocation(self): '''Get the geolocation associated with this status message Returns: The geolocation string of this status message. ''' return self._location def SetLocation(self, location): '''Set the geolocation associated with this status message Args: location: The geolocation string of this status message ''' self._location = location location = property(GetLocation, SetLocation, doc='The geolocation string of this status message') def GetRelativeCreatedAt(self): '''Get a human redable string representing the posting time Returns: A human readable string representing the posting time ''' fudge = 1.25 delta = long(self.now) - long(self.created_at_in_seconds) if delta < (1 * fudge): return 'about a second ago' elif delta < (60 * (1/fudge)): return 'about %d seconds ago' % (delta) elif delta < (60 * fudge): return 'about a minute ago' elif delta < (60 * 60 * (1/fudge)): return 'about %d minutes ago' % (delta / 60) elif delta < (60 * 60 * fudge) or delta / (60 * 60) == 1: return 'about an hour ago' elif delta < (60 * 60 * 24 * (1/fudge)): return 'about %d hours ago' % (delta / (60 * 60)) elif delta < (60 * 60 * 24 * fudge) or delta / (60 * 60 * 24) == 1: return 'about a day ago' else: return 'about %d days ago' % (delta / (60 * 60 * 24)) relative_created_at = property(GetRelativeCreatedAt, doc='Get a human readable string representing ' 'the posting time') def GetUser(self): '''Get a twitter.User reprenting the entity posting this status message. Returns: A twitter.User reprenting the entity posting this status message ''' return self._user def SetUser(self, user): '''Set a twitter.User reprenting the entity posting this status message. Args: user: A twitter.User reprenting the entity posting this status message ''' self._user = user user = property(GetUser, SetUser, doc='A twitter.User reprenting the entity posting this ' 'status message') def GetNow(self): '''Get the wallclock time for this status message. Used to calculate relative_created_at. Defaults to the time the object was instantiated. Returns: Whatever the status instance believes the current time to be, in seconds since the epoch. ''' if self._now is None: self._now = time.time() return self._now def SetNow(self, now): '''Set the wallclock time for this status message. Used to calculate relative_created_at. Defaults to the time the object was instantiated. Args: now: The wallclock time for this instance. ''' self._now = now now = property(GetNow, SetNow, doc='The wallclock time for this status instance.') def GetGeo(self): return self._geo def SetGeo(self, geo): self._geo = geo geo = property(GetGeo, SetGeo, doc='') def GetPlace(self): return self._place def SetPlace(self, place): self._place = place place = property(GetPlace, SetPlace, doc='') def GetCoordinates(self): return self._coordinates def SetCoordinates(self, coordinates): self._coordinates = coordinates coordinates = property(GetCoordinates, SetCoordinates, doc='') def GetContributors(self): return self._contributors def SetContributors(self, contributors): self._contributors = contributors contributors = property(GetContributors, SetContributors, doc='') def GetRetweeted_status(self): return self._retweeted_status def SetRetweeted_status(self, retweeted_status): self._retweeted_status = retweeted_status retweeted_status = property(GetRetweeted_status, SetRetweeted_status, doc='') def GetRetweetCount(self): return self._retweet_count def SetRetweetCount(self, retweet_count): self._retweet_count = retweet_count retweet_count = property(GetRetweetCount, SetRetweetCount, doc='') def __ne__(self, other): return not self.__eq__(other) def __eq__(self, other): try: return other and \ self.created_at == other.created_at and \ self.id == other.id and \ self.text == other.text and \ self.location == other.location and \ self.user == other.user and \ self.in_reply_to_screen_name == other.in_reply_to_screen_name and \ self.in_reply_to_user_id == other.in_reply_to_user_id and \ self.in_reply_to_status_id == other.in_reply_to_status_id and \ self.truncated == other.truncated and \ self.retweeted == other.retweeted and \ self.favorited == other.favorited and \ self.source == other.source and \ self.geo == other.geo and \ self.place == other.place and \ self.coordinates == other.coordinates and \ self.contributors == other.contributors and \ self.retweeted_status == other.retweeted_status and \ self.retweet_count == other.retweet_count except AttributeError: return False def __str__(self): '''A string representation of this twitter.Status instance. The return value is the same as the JSON string representation. Returns: A string representation of this twitter.Status instance. ''' return self.AsJsonString() def AsJsonString(self): '''A JSON string representation of this twitter.Status instance. Returns: A JSON string representation of this twitter.Status instance ''' return simplejson.dumps(self.AsDict(), sort_keys=True) def AsDict(self): '''A dict representation of this twitter.Status instance. The return value uses the same key names as the JSON representation. Return: A dict representing this twitter.Status instance ''' data = {} if self.created_at: data['created_at'] = self.created_at if self.favorited: data['favorited'] = self.favorited if self.id: data['id'] = self.id if self.text: data['text'] = self.text if self.location: data['location'] = self.location if self.user: data['user'] = self.user.AsDict() if self.in_reply_to_screen_name: data['in_reply_to_screen_name'] = self.in_reply_to_screen_name if self.in_reply_to_user_id: data['in_reply_to_user_id'] = self.in_reply_to_user_id if self.in_reply_to_status_id: data['in_reply_to_status_id'] = self.in_reply_to_status_id if self.truncated is not None: data['truncated'] = self.truncated if self.retweeted is not None: data['retweeted'] = self.retweeted if self.favorited is not None: data['favorited'] = self.favorited if self.source: data['source'] = self.source if self.geo: data['geo'] = self.geo if self.place: data['place'] = self.place if self.coordinates: data['coordinates'] = self.coordinates if self.contributors: data['contributors'] = self.contributors if self.hashtags: data['hashtags'] = [h.text for h in self.hashtags] if self.retweeted_status: data['retweeted_status'] = self.retweeted_status.AsDict() if self.retweet_count: data['retweet_count'] = self.retweet_count return data @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.Status instance ''' if 'user' in data: user = User.NewFromJsonDict(data['user']) else: user = None if 'retweeted_status' in data: retweeted_status = Status.NewFromJsonDict(data['retweeted_status']) else: retweeted_status = None urls = None user_mentions = None hashtags = None if 'entities' in data: if 'urls' in data['entities']: urls = [Url.NewFromJsonDict(u) for u in data['entities']['urls']] if 'user_mentions' in data['entities']: user_mentions = [User.NewFromJsonDict(u) for u in data['entities']['user_mentions']] if 'hashtags' in data['entities']: hashtags = [Hashtag.NewFromJsonDict(h) for h in data['entities']['hashtags']] return Status(created_at=data.get('created_at', None), favorited=data.get('favorited', None), id=data.get('id', None), text=data.get('text', None), location=data.get('location', None), in_reply_to_screen_name=data.get('in_reply_to_screen_name', None), in_reply_to_user_id=data.get('in_reply_to_user_id', None), in_reply_to_status_id=data.get('in_reply_to_status_id', None), truncated=data.get('truncated', None), retweeted=data.get('retweeted', None), source=data.get('source', None), user=user, urls=urls, user_mentions=user_mentions, hashtags=hashtags, geo=data.get('geo', None), place=data.get('place', None), coordinates=data.get('coordinates', None), contributors=data.get('contributors', None), retweeted_status=retweeted_status, retweet_count=data.get('retweet_count', None)) class User(object): '''A class representing the User structure used by the twitter API. The User structure exposes the following properties: user.id user.name user.screen_name user.location user.description user.profile_image_url user.profile_background_tile user.profile_background_image_url user.profile_sidebar_fill_color user.profile_background_color user.profile_link_color user.profile_text_color user.protected user.utc_offset user.time_zone user.url user.status user.statuses_count user.followers_count user.friends_count user.favourites_count user.geo_enabled user.verified user.lang user.notifications user.contributors_enabled user.created_at user.listed_count ''' def __init__(self, id=None, name=None, screen_name=None, location=None, description=None, profile_image_url=None, profile_background_tile=None, profile_background_image_url=None, profile_sidebar_fill_color=None, profile_background_color=None, profile_link_color=None, profile_text_color=None, protected=None, utc_offset=None, time_zone=None, followers_count=None, friends_count=None, statuses_count=None, favourites_count=None, url=None, status=None, geo_enabled=None, verified=None, lang=None, notifications=None, contributors_enabled=None, created_at=None, listed_count=None): self.id = id self.name = name self.screen_name = screen_name self.location = location self.description = description self.profile_image_url = profile_image_url self.profile_background_tile = profile_background_tile self.profile_background_image_url = profile_background_image_url self.profile_sidebar_fill_color = profile_sidebar_fill_color self.profile_background_color = profile_background_color self.profile_link_color = profile_link_color self.profile_text_color = profile_text_color self.protected = protected self.utc_offset = utc_offset self.time_zone = time_zone self.followers_count = followers_count self.friends_count = friends_count self.statuses_count = statuses_count self.favourites_count = favourites_count self.url = url self.status = status self.geo_enabled = geo_enabled self.verified = verified self.lang = lang self.notifications = notifications self.contributors_enabled = contributors_enabled self.created_at = created_at self.listed_count = listed_count def GetId(self): '''Get the unique id of this user. Returns: The unique id of this user ''' return self._id def SetId(self, id): '''Set the unique id of this user. Args: id: The unique id of this user. ''' self._id = id id = property(GetId, SetId, doc='The unique id of this user.') def GetName(self): '''Get the real name of this user. Returns: The real name of this user ''' return self._name def SetName(self, name): '''Set the real name of this user. Args: name: The real name of this user ''' self._name = name name = property(GetName, SetName, doc='The real name of this user.') def GetScreenName(self): '''Get the short twitter name of this user. Returns: The short twitter name of this user ''' return self._screen_name def SetScreenName(self, screen_name): '''Set the short twitter name of this user. Args: screen_name: the short twitter name of this user ''' self._screen_name = screen_name screen_name = property(GetScreenName, SetScreenName, doc='The short twitter name of this user.') def GetLocation(self): '''Get the geographic location of this user. Returns: The geographic location of this user ''' return self._location def SetLocation(self, location): '''Set the geographic location of this user. Args: location: The geographic location of this user ''' self._location = location location = property(GetLocation, SetLocation, doc='The geographic location of this user.') def GetDescription(self): '''Get the short text description of this user. Returns: The short text description of this user ''' return self._description def SetDescription(self, description): '''Set the short text description of this user. Args: description: The short text description of this user ''' self._description = description description = property(GetDescription, SetDescription, doc='The short text description of this user.') def GetUrl(self): '''Get the homepage url of this user. Returns: The homepage url of this user ''' return self._url def SetUrl(self, url): '''Set the homepage url of this user. Args: url: The homepage url of this user ''' self._url = url url = property(GetUrl, SetUrl, doc='The homepage url of this user.') def GetProfileImageUrl(self): '''Get the url of the thumbnail of this user. Returns: The url of the thumbnail of this user ''' return self._profile_image_url def SetProfileImageUrl(self, profile_image_url): '''Set the url of the thumbnail of this user. Args: profile_image_url: The url of the thumbnail of this user ''' self._profile_image_url = profile_image_url profile_image_url= property(GetProfileImageUrl, SetProfileImageUrl, doc='The url of the thumbnail of this user.') def GetProfileBackgroundTile(self): '''Boolean for whether to tile the profile background image. Returns: True if the background is to be tiled, False if not, None if unset. ''' return self._profile_background_tile def SetProfileBackgroundTile(self, profile_background_tile): '''Set the boolean flag for whether to tile the profile background image. Args: profile_background_tile: Boolean flag for whether to tile or not. ''' self._profile_background_tile = profile_background_tile profile_background_tile = property(GetProfileBackgroundTile, SetProfileBackgroundTile, doc='Boolean for whether to tile the background image.') def GetProfileBackgroundImageUrl(self): return self._profile_background_image_url def SetProfileBackgroundImageUrl(self, profile_background_image_url): self._profile_background_image_url = profile_background_image_url profile_background_image_url = property(GetProfileBackgroundImageUrl, SetProfileBackgroundImageUrl, doc='The url of the profile background of this user.') def GetProfileSidebarFillColor(self): return self._profile_sidebar_fill_color def SetProfileSidebarFillColor(self, profile_sidebar_fill_color): self._profile_sidebar_fill_color = profile_sidebar_fill_color profile_sidebar_fill_color = property(GetProfileSidebarFillColor, SetProfileSidebarFillColor) def GetProfileBackgroundColor(self): return self._profile_background_color def SetProfileBackgroundColor(self, profile_background_color): self._profile_background_color = profile_background_color profile_background_color = property(GetProfileBackgroundColor, SetProfileBackgroundColor) def GetProfileLinkColor(self): return self._profile_link_color def SetProfileLinkColor(self, profile_link_color): self._profile_link_color = profile_link_color profile_link_color = property(GetProfileLinkColor, SetProfileLinkColor) def GetProfileTextColor(self): return self._profile_text_color def SetProfileTextColor(self, profile_text_color): self._profile_text_color = profile_text_color profile_text_color = property(GetProfileTextColor, SetProfileTextColor) def GetProtected(self): return self._protected def SetProtected(self, protected): self._protected = protected protected = property(GetProtected, SetProtected) def GetUtcOffset(self): return self._utc_offset def SetUtcOffset(self, utc_offset): self._utc_offset = utc_offset utc_offset = property(GetUtcOffset, SetUtcOffset) def GetTimeZone(self): '''Returns the current time zone string for the user. Returns: The descriptive time zone string for the user. ''' return self._time_zone def SetTimeZone(self, time_zone): '''Sets the user's time zone string. Args: time_zone: The descriptive time zone to assign for the user. ''' self._time_zone = time_zone time_zone = property(GetTimeZone, SetTimeZone) def GetStatus(self): '''Get the latest twitter.Status of this user. Returns: The latest twitter.Status of this user ''' return self._status def SetStatus(self, status): '''Set the latest twitter.Status of this user. Args: status: The latest twitter.Status of this user ''' self._status = status status = property(GetStatus, SetStatus, doc='The latest twitter.Status of this user.') def GetFriendsCount(self): '''Get the friend count for this user. Returns: The number of users this user has befriended. ''' return self._friends_count def SetFriendsCount(self, count): '''Set the friend count for this user. Args: count: The number of users this user has befriended. ''' self._friends_count = count friends_count = property(GetFriendsCount, SetFriendsCount, doc='The number of friends for this user.') def GetListedCount(self): '''Get the listed count for this user. Returns: The number of lists this user belongs to. ''' return self._listed_count def SetListedCount(self, count): '''Set the listed count for this user. Args: count: The number of lists this user belongs to. ''' self._listed_count = count listed_count = property(GetListedCount, SetListedCount, doc='The number of lists this user belongs to.') def GetFollowersCount(self): '''Get the follower count for this user. Returns: The number of users following this user. ''' return self._followers_count def SetFollowersCount(self, count): '''Set the follower count for this user. Args: count: The number of users following this user. ''' self._followers_count = count followers_count = property(GetFollowersCount, SetFollowersCount, doc='The number of users following this user.') def GetStatusesCount(self): '''Get the number of status updates for this user. Returns: The number of status updates for this user. ''' return self._statuses_count def SetStatusesCount(self, count): '''Set the status update count for this user. Args: count: The number of updates for this user. ''' self._statuses_count = count statuses_count = property(GetStatusesCount, SetStatusesCount, doc='The number of updates for this user.') def GetFavouritesCount(self): '''Get the number of favourites for this user. Returns: The number of favourites for this user. ''' return self._favourites_count def SetFavouritesCount(self, count): '''Set the favourite count for this user. Args: count: The number of favourites for this user. ''' self._favourites_count = count favourites_count = property(GetFavouritesCount, SetFavouritesCount, doc='The number of favourites for this user.') def GetGeoEnabled(self): '''Get the setting of geo_enabled for this user. Returns: True/False if Geo tagging is enabled ''' return self._geo_enabled def SetGeoEnabled(self, geo_enabled): '''Set the latest twitter.geo_enabled of this user. Args: geo_enabled: True/False if Geo tagging is to be enabled ''' self._geo_enabled = geo_enabled geo_enabled = property(GetGeoEnabled, SetGeoEnabled, doc='The value of twitter.geo_enabled for this user.') def GetVerified(self): '''Get the setting of verified for this user. Returns: True/False if user is a verified account ''' return self._verified def SetVerified(self, verified): '''Set twitter.verified for this user. Args: verified: True/False if user is a verified account ''' self._verified = verified verified = property(GetVerified, SetVerified, doc='The value of twitter.verified for this user.') def GetLang(self): '''Get the setting of lang for this user. Returns: language code of the user ''' return self._lang def SetLang(self, lang): '''Set twitter.lang for this user. Args: lang: language code for the user ''' self._lang = lang lang = property(GetLang, SetLang, doc='The value of twitter.lang for this user.') def GetNotifications(self): '''Get the setting of notifications for this user. Returns: True/False for the notifications setting of the user ''' return self._notifications def SetNotifications(self, notifications): '''Set twitter.notifications for this user. Args: notifications: True/False notifications setting for the user ''' self._notifications = notifications notifications = property(GetNotifications, SetNotifications, doc='The value of twitter.notifications for this user.') def GetContributorsEnabled(self): '''Get the setting of contributors_enabled for this user. Returns: True/False contributors_enabled of the user ''' return self._contributors_enabled def SetContributorsEnabled(self, contributors_enabled): '''Set twitter.contributors_enabled for this user. Args: contributors_enabled: True/False contributors_enabled setting for the user ''' self._contributors_enabled = contributors_enabled contributors_enabled = property(GetContributorsEnabled, SetContributorsEnabled, doc='The value of twitter.contributors_enabled for this user.') def GetCreatedAt(self): '''Get the setting of created_at for this user. Returns: created_at value of the user ''' return self._created_at def SetCreatedAt(self, created_at): '''Set twitter.created_at for this user. Args: created_at: created_at value for the user ''' self._created_at = created_at created_at = property(GetCreatedAt, SetCreatedAt, doc='The value of twitter.created_at for this user.') def __ne__(self, other): return not self.__eq__(other) def __eq__(self, other): try: return other and \ self.id == other.id and \ self.name == other.name and \ self.screen_name == other.screen_name and \ self.location == other.location and \ self.description == other.description and \ self.profile_image_url == other.profile_image_url and \ self.profile_background_tile == other.profile_background_tile and \ self.profile_background_image_url == other.profile_background_image_url and \ self.profile_sidebar_fill_color == other.profile_sidebar_fill_color and \ self.profile_background_color == other.profile_background_color and \ self.profile_link_color == other.profile_link_color and \ self.profile_text_color == other.profile_text_color and \ self.protected == other.protected and \ self.utc_offset == other.utc_offset and \ self.time_zone == other.time_zone and \ self.url == other.url and \ self.statuses_count == other.statuses_count and \ self.followers_count == other.followers_count and \ self.favourites_count == other.favourites_count and \ self.friends_count == other.friends_count and \ self.status == other.status and \ self.geo_enabled == other.geo_enabled and \ self.verified == other.verified and \ self.lang == other.lang and \ self.notifications == other.notifications and \ self.contributors_enabled == other.contributors_enabled and \ self.created_at == other.created_at and \ self.listed_count == other.listed_count except AttributeError: return False def __str__(self): '''A string representation of this twitter.User instance. The return value is the same as the JSON string representation. Returns: A string representation of this twitter.User instance. ''' return self.AsJsonString() def AsJsonString(self): '''A JSON string representation of this twitter.User instance. Returns: A JSON string representation of this twitter.User instance ''' return simplejson.dumps(self.AsDict(), sort_keys=True) def AsDict(self): '''A dict representation of this twitter.User instance. The return value uses the same key names as the JSON representation. Return: A dict representing this twitter.User instance ''' data = {} if self.id: data['id'] = self.id if self.name: data['name'] = self.name if self.screen_name: data['screen_name'] = self.screen_name if self.location: data['location'] = self.location if self.description: data['description'] = self.description if self.profile_image_url: data['profile_image_url'] = self.profile_image_url if self.profile_background_tile is not None: data['profile_background_tile'] = self.profile_background_tile if self.profile_background_image_url: data['profile_sidebar_fill_color'] = self.profile_background_image_url if self.profile_background_color: data['profile_background_color'] = self.profile_background_color if self.profile_link_color: data['profile_link_color'] = self.profile_link_color if self.profile_text_color: data['profile_text_color'] = self.profile_text_color if self.protected is not None: data['protected'] = self.protected if self.utc_offset: data['utc_offset'] = self.utc_offset if self.time_zone: data['time_zone'] = self.time_zone if self.url: data['url'] = self.url if self.status: data['status'] = self.status.AsDict() if self.friends_count: data['friends_count'] = self.friends_count if self.followers_count: data['followers_count'] = self.followers_count if self.statuses_count: data['statuses_count'] = self.statuses_count if self.favourites_count: data['favourites_count'] = self.favourites_count if self.geo_enabled: data['geo_enabled'] = self.geo_enabled if self.verified: data['verified'] = self.verified if self.lang: data['lang'] = self.lang if self.notifications: data['notifications'] = self.notifications if self.contributors_enabled: data['contributors_enabled'] = self.contributors_enabled if self.created_at: data['created_at'] = self.created_at if self.listed_count: data['listed_count'] = self.listed_count return data @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.User instance ''' if 'status' in data: status = Status.NewFromJsonDict(data['status']) else: status = None return User(id=data.get('id', None), name=data.get('name', None), screen_name=data.get('screen_name', None), location=data.get('location', None), description=data.get('description', None), statuses_count=data.get('statuses_count', None), followers_count=data.get('followers_count', None), favourites_count=data.get('favourites_count', None), friends_count=data.get('friends_count', None), profile_image_url=data.get('profile_image_url', None), profile_background_tile = data.get('profile_background_tile', None), profile_background_image_url = data.get('profile_background_image_url', None), profile_sidebar_fill_color = data.get('profile_sidebar_fill_color', None), profile_background_color = data.get('profile_background_color', None), profile_link_color = data.get('profile_link_color', None), profile_text_color = data.get('profile_text_color', None), protected = data.get('protected', None), utc_offset = data.get('utc_offset', None), time_zone = data.get('time_zone', None), url=data.get('url', None), status=status, geo_enabled=data.get('geo_enabled', None), verified=data.get('verified', None), lang=data.get('lang', None), notifications=data.get('notifications', None), contributors_enabled=data.get('contributors_enabled', None), created_at=data.get('created_at', None), listed_count=data.get('listed_count', None)) class List(object): '''A class representing the List structure used by the twitter API. The List structure exposes the following properties: list.id list.name list.slug list.description list.full_name list.mode list.uri list.member_count list.subscriber_count list.following ''' def __init__(self, id=None, name=None, slug=None, description=None, full_name=None, mode=None, uri=None, member_count=None, subscriber_count=None, following=None, user=None): self.id = id self.name = name self.slug = slug self.description = description self.full_name = full_name self.mode = mode self.uri = uri self.member_count = member_count self.subscriber_count = subscriber_count self.following = following self.user = user def GetId(self): '''Get the unique id of this list. Returns: The unique id of this list ''' return self._id def SetId(self, id): '''Set the unique id of this list. Args: id: The unique id of this list. ''' self._id = id id = property(GetId, SetId, doc='The unique id of this list.') def GetName(self): '''Get the real name of this list. Returns: The real name of this list ''' return self._name def SetName(self, name): '''Set the real name of this list. Args: name: The real name of this list ''' self._name = name name = property(GetName, SetName, doc='The real name of this list.') def GetSlug(self): '''Get the slug of this list. Returns: The slug of this list ''' return self._slug def SetSlug(self, slug): '''Set the slug of this list. Args: slug: The slug of this list. ''' self._slug = slug slug = property(GetSlug, SetSlug, doc='The slug of this list.') def GetDescription(self): '''Get the description of this list. Returns: The description of this list ''' return self._description def SetDescription(self, description): '''Set the description of this list. Args: description: The description of this list. ''' self._description = description description = property(GetDescription, SetDescription, doc='The description of this list.') def GetFull_name(self): '''Get the full_name of this list. Returns: The full_name of this list ''' return self._full_name def SetFull_name(self, full_name): '''Set the full_name of this list. Args: full_name: The full_name of this list. ''' self._full_name = full_name full_name = property(GetFull_name, SetFull_name, doc='The full_name of this list.') def GetMode(self): '''Get the mode of this list. Returns: The mode of this list ''' return self._mode def SetMode(self, mode): '''Set the mode of this list. Args: mode: The mode of this list. ''' self._mode = mode mode = property(GetMode, SetMode, doc='The mode of this list.') def GetUri(self): '''Get the uri of this list. Returns: The uri of this list ''' return self._uri def SetUri(self, uri): '''Set the uri of this list. Args: uri: The uri of this list. ''' self._uri = uri uri = property(GetUri, SetUri, doc='The uri of this list.') def GetMember_count(self): '''Get the member_count of this list. Returns: The member_count of this list ''' return self._member_count def SetMember_count(self, member_count): '''Set the member_count of this list. Args: member_count: The member_count of this list. ''' self._member_count = member_count member_count = property(GetMember_count, SetMember_count, doc='The member_count of this list.') def GetSubscriber_count(self): '''Get the subscriber_count of this list. Returns: The subscriber_count of this list ''' return self._subscriber_count def SetSubscriber_count(self, subscriber_count): '''Set the subscriber_count of this list. Args: subscriber_count: The subscriber_count of this list. ''' self._subscriber_count = subscriber_count subscriber_count = property(GetSubscriber_count, SetSubscriber_count, doc='The subscriber_count of this list.') def GetFollowing(self): '''Get the following status of this list. Returns: The following status of this list ''' return self._following def SetFollowing(self, following): '''Set the following status of this list. Args: following: The following of this list. ''' self._following = following following = property(GetFollowing, SetFollowing, doc='The following status of this list.') def GetUser(self): '''Get the user of this list. Returns: The owner of this list ''' return self._user def SetUser(self, user): '''Set the user of this list. Args: user: The owner of this list. ''' self._user = user user = property(GetUser, SetUser, doc='The owner of this list.') def __ne__(self, other): return not self.__eq__(other) def __eq__(self, other): try: return other and \ self.id == other.id and \ self.name == other.name and \ self.slug == other.slug and \ self.description == other.description and \ self.full_name == other.full_name and \ self.mode == other.mode and \ self.uri == other.uri and \ self.member_count == other.member_count and \ self.subscriber_count == other.subscriber_count and \ self.following == other.following and \ self.user == other.user except AttributeError: return False def __str__(self): '''A string representation of this twitter.List instance. The return value is the same as the JSON string representation. Returns: A string representation of this twitter.List instance. ''' return self.AsJsonString() def AsJsonString(self): '''A JSON string representation of this twitter.List instance. Returns: A JSON string representation of this twitter.List instance ''' return simplejson.dumps(self.AsDict(), sort_keys=True) def AsDict(self): '''A dict representation of this twitter.List instance. The return value uses the same key names as the JSON representation. Return: A dict representing this twitter.List instance ''' data = {} if self.id: data['id'] = self.id if self.name: data['name'] = self.name if self.slug: data['slug'] = self.slug if self.description: data['description'] = self.description if self.full_name: data['full_name'] = self.full_name if self.mode: data['mode'] = self.mode if self.uri: data['uri'] = self.uri if self.member_count is not None: data['member_count'] = self.member_count if self.subscriber_count is not None: data['subscriber_count'] = self.subscriber_count if self.following is not None: data['following'] = self.following if self.user is not None: data['user'] = self.user return data @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.List instance ''' if 'user' in data: user = User.NewFromJsonDict(data['user']) else: user = None return List(id=data.get('id', None), name=data.get('name', None), slug=data.get('slug', None), description=data.get('description', None), full_name=data.get('full_name', None), mode=data.get('mode', None), uri=data.get('uri', None), member_count=data.get('member_count', None), subscriber_count=data.get('subscriber_count', None), following=data.get('following', None), user=user) class DirectMessage(object): '''A class representing the DirectMessage structure used by the twitter API. The DirectMessage structure exposes the following properties: direct_message.id direct_message.created_at direct_message.created_at_in_seconds # read only direct_message.sender_id direct_message.sender_screen_name direct_message.recipient_id direct_message.recipient_screen_name direct_message.text ''' def __init__(self, id=None, created_at=None, sender_id=None, sender_screen_name=None, recipient_id=None, recipient_screen_name=None, text=None): '''An object to hold a Twitter direct message. This class is normally instantiated by the twitter.Api class and returned in a sequence. Note: Dates are posted in the form "Sat Jan 27 04:17:38 +0000 2007" Args: id: The unique id of this direct message. [Optional] created_at: The time this direct message was posted. [Optional] sender_id: The id of the twitter user that sent this message. [Optional] sender_screen_name: The name of the twitter user that sent this message. [Optional] recipient_id: The id of the twitter that received this message. [Optional] recipient_screen_name: The name of the twitter that received this message. [Optional] text: The text of this direct message. [Optional] ''' self.id = id self.created_at = created_at self.sender_id = sender_id self.sender_screen_name = sender_screen_name self.recipient_id = recipient_id self.recipient_screen_name = recipient_screen_name self.text = text def GetId(self): '''Get the unique id of this direct message. Returns: The unique id of this direct message ''' return self._id def SetId(self, id): '''Set the unique id of this direct message. Args: id: The unique id of this direct message ''' self._id = id id = property(GetId, SetId, doc='The unique id of this direct message.') def GetCreatedAt(self): '''Get the time this direct message was posted. Returns: The time this direct message was posted ''' return self._created_at def SetCreatedAt(self, created_at): '''Set the time this direct message was posted. Args: created_at: The time this direct message was created ''' self._created_at = created_at created_at = property(GetCreatedAt, SetCreatedAt, doc='The time this direct message was posted.') def GetCreatedAtInSeconds(self): '''Get the time this direct message was posted, in seconds since the epoch. Returns: The time this direct message was posted, in seconds since the epoch. ''' return calendar.timegm(rfc822.parsedate(self.created_at)) created_at_in_seconds = property(GetCreatedAtInSeconds, doc="The time this direct message was " "posted, in seconds since the epoch") def GetSenderId(self): '''Get the unique sender id of this direct message. Returns: The unique sender id of this direct message ''' return self._sender_id def SetSenderId(self, sender_id): '''Set the unique sender id of this direct message. Args: sender_id: The unique sender id of this direct message ''' self._sender_id = sender_id sender_id = property(GetSenderId, SetSenderId, doc='The unique sender id of this direct message.') def GetSenderScreenName(self): '''Get the unique sender screen name of this direct message. Returns: The unique sender screen name of this direct message ''' return self._sender_screen_name def SetSenderScreenName(self, sender_screen_name): '''Set the unique sender screen name of this direct message. Args: sender_screen_name: The unique sender screen name of this direct message ''' self._sender_screen_name = sender_screen_name sender_screen_name = property(GetSenderScreenName, SetSenderScreenName, doc='The unique sender screen name of this direct message.') def GetRecipientId(self): '''Get the unique recipient id of this direct message. Returns: The unique recipient id of this direct message ''' return self._recipient_id def SetRecipientId(self, recipient_id): '''Set the unique recipient id of this direct message. Args: recipient_id: The unique recipient id of this direct message ''' self._recipient_id = recipient_id recipient_id = property(GetRecipientId, SetRecipientId, doc='The unique recipient id of this direct message.') def GetRecipientScreenName(self): '''Get the unique recipient screen name of this direct message. Returns: The unique recipient screen name of this direct message ''' return self._recipient_screen_name def SetRecipientScreenName(self, recipient_screen_name): '''Set the unique recipient screen name of this direct message. Args: recipient_screen_name: The unique recipient screen name of this direct message ''' self._recipient_screen_name = recipient_screen_name recipient_screen_name = property(GetRecipientScreenName, SetRecipientScreenName, doc='The unique recipient screen name of this direct message.') def GetText(self): '''Get the text of this direct message. Returns: The text of this direct message. ''' return self._text def SetText(self, text): '''Set the text of this direct message. Args: text: The text of this direct message ''' self._text = text text = property(GetText, SetText, doc='The text of this direct message') def __ne__(self, other): return not self.__eq__(other) def __eq__(self, other): try: return other and \ self.id == other.id and \ self.created_at == other.created_at and \ self.sender_id == other.sender_id and \ self.sender_screen_name == other.sender_screen_name and \ self.recipient_id == other.recipient_id and \ self.recipient_screen_name == other.recipient_screen_name and \ self.text == other.text except AttributeError: return False def __str__(self): '''A string representation of this twitter.DirectMessage instance. The return value is the same as the JSON string representation. Returns: A string representation of this twitter.DirectMessage instance. ''' return self.AsJsonString() def AsJsonString(self): '''A JSON string representation of this twitter.DirectMessage instance. Returns: A JSON string representation of this twitter.DirectMessage instance ''' return simplejson.dumps(self.AsDict(), sort_keys=True) def AsDict(self): '''A dict representation of this twitter.DirectMessage instance. The return value uses the same key names as the JSON representation. Return: A dict representing this twitter.DirectMessage instance ''' data = {} if self.id: data['id'] = self.id if self.created_at: data['created_at'] = self.created_at if self.sender_id: data['sender_id'] = self.sender_id if self.sender_screen_name: data['sender_screen_name'] = self.sender_screen_name if self.recipient_id: data['recipient_id'] = self.recipient_id if self.recipient_screen_name: data['recipient_screen_name'] = self.recipient_screen_name if self.text: data['text'] = self.text return data @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.DirectMessage instance ''' return DirectMessage(created_at=data.get('created_at', None), recipient_id=data.get('recipient_id', None), sender_id=data.get('sender_id', None), text=data.get('text', None), sender_screen_name=data.get('sender_screen_name', None), id=data.get('id', None), recipient_screen_name=data.get('recipient_screen_name', None)) class Hashtag(object): ''' A class represeinting a twitter hashtag ''' def __init__(self, text=None): self.text = text @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.Hashtag instance ''' return Hashtag(text = data.get('text', None)) class Trend(object): ''' A class representing a trending topic ''' def __init__(self, name=None, query=None, timestamp=None): self.name = name self.query = query self.timestamp = timestamp def __str__(self): return 'Name: %s\nQuery: %s\nTimestamp: %s\n' % (self.name, self.query, self.timestamp) def __ne__(self, other): return not self.__eq__(other) def __eq__(self, other): try: return other and \ self.name == other.name and \ self.query == other.query and \ self.timestamp == other.timestamp except AttributeError: return False @staticmethod def NewFromJsonDict(data, timestamp = None): '''Create a new instance based on a JSON dict Args: data: A JSON dict timestamp: Gets set as the timestamp property of the new object Returns: A twitter.Trend object ''' return Trend(name=data.get('name', None), query=data.get('query', None), timestamp=timestamp) class Url(object): '''A class representing an URL contained in a tweet''' def __init__(self, url=None, expanded_url=None): self.url = url self.expanded_url = expanded_url @staticmethod def NewFromJsonDict(data): '''Create a new instance based on a JSON dict. Args: data: A JSON dict, as converted from the JSON in the twitter API Returns: A twitter.Url instance ''' return Url(url=data.get('url', None), expanded_url=data.get('expanded_url', None)) class Api(object): '''A python interface into the Twitter API By default, the Api caches results for 1 minute. Example usage: To create an instance of the twitter.Api class, with no authentication: >>> import twitter >>> api = twitter.Api() To fetch the most recently posted public twitter status messages: >>> statuses = api.GetPublicTimeline() >>> print [s.user.name for s in statuses] [u'DeWitt', u'Kesuke Miyagi', u'ev', u'Buzz Andersen', u'Biz Stone'] #... To fetch a single user's public status messages, where "user" is either a Twitter "short name" or their user id. >>> statuses = api.GetUserTimeline(user) >>> print [s.text for s in statuses] To use authentication, instantiate the twitter.Api class with a consumer key and secret; and the oAuth key and secret: >>> api = twitter.Api(consumer_key='twitter consumer key', consumer_secret='twitter consumer secret', access_token_key='the_key_given', access_token_secret='the_key_secret') To fetch your friends (after being authenticated): >>> users = api.GetFriends() >>> print [u.name for u in users] To post a twitter status message (after being authenticated): >>> status = api.PostUpdate('I love python-twitter!') >>> print status.text I love python-twitter! There are many other methods, including: >>> api.PostUpdates(status) >>> api.PostDirectMessage(user, text) >>> api.GetUser(user) >>> api.GetReplies() >>> api.GetUserTimeline(user) >>> api.GetStatus(id) >>> api.DestroyStatus(id) >>> api.GetFriendsTimeline(user) >>> api.GetFriends(user) >>> api.GetFollowers() >>> api.GetFeatured() >>> api.GetDirectMessages() >>> api.PostDirectMessage(user, text) >>> api.DestroyDirectMessage(id) >>> api.DestroyFriendship(user) >>> api.CreateFriendship(user) >>> api.GetUserByEmail(email) >>> api.VerifyCredentials() ''' DEFAULT_CACHE_TIMEOUT = 60 # cache for 1 minute _API_REALM = 'Twitter API' def __init__(self, consumer_key=None, consumer_secret=None, access_token_key=None, access_token_secret=None, input_encoding=None, request_headers=None, cache=DEFAULT_CACHE, shortner=None, base_url=None, use_gzip_compression=False, debugHTTP=False): '''Instantiate a new twitter.Api object. Args: consumer_key: Your Twitter user's consumer_key. consumer_secret: Your Twitter user's consumer_secret. access_token_key: The oAuth access token key value you retrieved from running get_access_token.py. access_token_secret: The oAuth access token's secret, also retrieved from the get_access_token.py run. input_encoding: The encoding used to encode input strings. [Optional] request_header: A dictionary of additional HTTP request headers. [Optional] cache: The cache instance to use. Defaults to DEFAULT_CACHE. Use None to disable caching. [Optional] shortner: The shortner instance to use. Defaults to None. See shorten_url.py for an example shortner. [Optional] base_url: The base URL to use to contact the Twitter API. Defaults to https://twitter.com. [Optional] use_gzip_compression: Set to True to tell enable gzip compression for any call made to Twitter. Defaults to False. [Optional] debugHTTP: Set to True to enable debug output from urllib2 when performing any HTTP requests. Defaults to False. [Optional] ''' self.SetCache(cache) self._urllib = urllib2 self._cache_timeout = Api.DEFAULT_CACHE_TIMEOUT self._input_encoding = input_encoding self._use_gzip = use_gzip_compression self._debugHTTP = debugHTTP self._oauth_consumer = None self._InitializeRequestHeaders(request_headers) self._InitializeUserAgent() self._InitializeDefaultParameters() if base_url is None: self.base_url = 'https://api.twitter.com/1' else: self.base_url = base_url if consumer_key is not None and (access_token_key is None or access_token_secret is None): print >> sys.stderr, 'Twitter now requires an oAuth Access Token for API calls.' print >> sys.stderr, 'If your using this library from a command line utility, please' print >> sys.stderr, 'run the the included get_access_token.py tool to generate one.' raise TwitterError('Twitter requires oAuth Access Token for all API access') self.SetCredentials(consumer_key, consumer_secret, access_token_key, access_token_secret) def SetCredentials(self, consumer_key, consumer_secret, access_token_key=None, access_token_secret=None): '''Set the consumer_key and consumer_secret for this instance Args: consumer_key: The consumer_key of the twitter account. consumer_secret: The consumer_secret for the twitter account. access_token_key: The oAuth access token key value you retrieved from running get_access_token.py. access_token_secret: The oAuth access token's secret, also retrieved from the get_access_token.py run. ''' self._consumer_key = consumer_key self._consumer_secret = consumer_secret self._access_token_key = access_token_key self._access_token_secret = access_token_secret self._oauth_consumer = None if consumer_key is not None and consumer_secret is not None and \ access_token_key is not None and access_token_secret is not None: self._signature_method_plaintext = oauth.SignatureMethod_PLAINTEXT() self._signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1() self._oauth_token = oauth.Token(key=access_token_key, secret=access_token_secret) self._oauth_consumer = oauth.Consumer(key=consumer_key, secret=consumer_secret) def ClearCredentials(self): '''Clear the any credentials for this instance ''' self._consumer_key = None self._consumer_secret = None self._access_token_key = None self._access_token_secret = None self._oauth_consumer = None def GetPublicTimeline(self, since_id=None, include_rts=None, include_entities=None): '''Fetch the sequence of public twitter.Status message for all users. Args: since_id: Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available. [Optional] include_rts: If True, the timeline will contain native retweets (if they exist) in addition to the standard stream of tweets. [Optional] include_entities: If True, each tweet will include a node called "entities,". This node offers a variety of metadata about the tweet in a discreet structure, including: user_mentions, urls, and hashtags. [Optional] Returns: An sequence of twitter.Status instances, one for each message ''' parameters = {} if since_id: parameters['since_id'] = since_id if include_rts: parameters['include_rts'] = 1 if include_entities: parameters['include_entities'] = 1 url = '%s/statuses/public_timeline.json' % self.base_url json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) return [Status.NewFromJsonDict(x) for x in data] def FilterPublicTimeline(self, term, since_id=None): '''Filter the public twitter timeline by a given search term on the local machine. Args: term: term to search by. since_id: Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available. [Optional] Returns: A sequence of twitter.Status instances, one for each message containing the term ''' statuses = self.GetPublicTimeline(since_id) results = [] for s in statuses: if s.text.lower().find(term.lower()) != -1: results.append(s) return results def GetSearch(self, term=None, geocode=None, since_id=None, per_page=15, page=1, lang="en", show_user="true", query_users=False): '''Return twitter search results for a given term. Args: term: term to search by. Optional if you include geocode. since_id: Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available. [Optional] geocode: geolocation information in the form (latitude, longitude, radius) [Optional] per_page: number of results to return. Default is 15 [Optional] page: Specifies the page of results to retrieve. Note: there are pagination limits. [Optional] lang: language for results. Default is English [Optional] show_user: prefixes screen name in status query_users: If set to False, then all users only have screen_name and profile_image_url available. If set to True, all information of users are available, but it uses lots of request quota, one per status. Returns: A sequence of twitter.Status instances, one for each message containing the term ''' # Build request parameters parameters = {} if since_id: parameters['since_id'] = since_id if term is None and geocode is None: return [] if term is not None: parameters['q'] = term if geocode is not None: parameters['geocode'] = ','.join(map(str, geocode)) parameters['show_user'] = show_user parameters['lang'] = lang parameters['rpp'] = per_page parameters['page'] = page # Make and send requests url = 'http://search.twitter.com/search.json' json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) results = [] for x in data['results']: temp = Status.NewFromJsonDict(x) if query_users: # Build user object with new request temp.user = self.GetUser(urllib.quote(x['from_user'])) else: temp.user = User(screen_name=x['from_user'], profile_image_url=x['profile_image_url']) results.append(temp) # Return built list of statuses return results # [Status.NewFromJsonDict(x) for x in data['results']] def GetTrendsCurrent(self, exclude=None): '''Get the current top trending topics Args: exclude: Appends the exclude parameter as a request parameter. Currently only exclude=hashtags is supported. [Optional] Returns: A list with 10 entries. Each entry contains the twitter. ''' parameters = {} if exclude: parameters['exclude'] = exclude url = '%s/trends/current.json' % self.base_url json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) trends = [] for t in data['trends']: for item in data['trends'][t]: trends.append(Trend.NewFromJsonDict(item, timestamp = t)) return trends def GetTrendsWoeid(self, woeid, exclude=None): '''Return the top 10 trending topics for a specific WOEID, if trending information is available for it. Args: woeid: the Yahoo! Where On Earth ID for a location. exclude: Appends the exclude parameter as a request parameter. Currently only exclude=hashtags is supported. [Optional] Returns: A list with 10 entries. Each entry contains a Trend. ''' parameters = {} if exclude: parameters['exclude'] = exclude url = '%s/trends/%s.json' % (self.base_url, woeid) json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) trends = [] timestamp = data[0]['as_of'] for trend in data[0]['trends']: trends.append(Trend.NewFromJsonDict(trend, timestamp = timestamp)) return trends def GetTrendsDaily(self, exclude=None, startdate=None): '''Get the current top trending topics for each hour in a given day Args: startdate: The start date for the report. Should be in the format YYYY-MM-DD. [Optional] exclude: Appends the exclude parameter as a request parameter. Currently only exclude=hashtags is supported. [Optional] Returns: A list with 24 entries. Each entry contains the twitter. Trend elements that were trending at the corresponding hour of the day. ''' parameters = {} if exclude: parameters['exclude'] = exclude if not startdate: startdate = time.strftime('%Y-%m-%d', time.gmtime()) parameters['date'] = startdate url = '%s/trends/daily.json' % self.base_url json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) trends = [] for i in xrange(24): trends.append(None) for t in data['trends']: idx = int(time.strftime('%H', time.strptime(t, '%Y-%m-%d %H:%M'))) trends[idx] = [Trend.NewFromJsonDict(x, timestamp = t) for x in data['trends'][t]] return trends def GetTrendsWeekly(self, exclude=None, startdate=None): '''Get the top 30 trending topics for each day in a given week. Args: startdate: The start date for the report. Should be in the format YYYY-MM-DD. [Optional] exclude: Appends the exclude parameter as a request parameter. Currently only exclude=hashtags is supported. [Optional] Returns: A list with each entry contains the twitter. Trend elements of trending topics for the corrsponding day of the week ''' parameters = {} if exclude: parameters['exclude'] = exclude if not startdate: startdate = time.strftime('%Y-%m-%d', time.gmtime()) parameters['date'] = startdate url = '%s/trends/weekly.json' % self.base_url json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) trends = [] for i in xrange(7): trends.append(None) # use the epochs of the dates as keys for a dictionary times = dict([(calendar.timegm(time.strptime(t, '%Y-%m-%d')),t) for t in data['trends']]) cnt = 0 # create the resulting structure ordered by the epochs of the dates for e in sorted(times.keys()): trends[cnt] = [Trend.NewFromJsonDict(x, timestamp = times[e]) for x in data['trends'][times[e]]] cnt +=1 return trends def GetFriendsTimeline(self, user=None, count=None, page=None, since_id=None, retweets=None, include_entities=None): '''Fetch the sequence of twitter.Status messages for a user's friends The twitter.Api instance must be authenticated if the user is private. Args: user: Specifies the ID or screen name of the user for whom to return the friends_timeline. If not specified then the authenticated user set in the twitter.Api instance will be used. [Optional] count: Specifies the number of statuses to retrieve. May not be greater than 100. [Optional] page: Specifies the page of results to retrieve. Note: there are pagination limits. [Optional] since_id: Returns results with an ID greater than (that is, more recent than) the specified ID. There are limits to the number of Tweets which can be accessed through the API. If the limit of Tweets has occured since the since_id, the since_id will be forced to the oldest ID available. [Optional] retweets: If True, the timeline will contain native retweets. [Optional] include_entities: If True, each tweet will include a node called "entities,". This node offers a variety of metadata about the tweet in a discreet structure, including: user_mentions, urls, and hashtags. [Optional] Returns: A sequence of twitter.Status instances, one for each message ''' if not user and not self._oauth_consumer: raise TwitterError("User must be specified if API is not authenticated.") url = '%s/statuses/friends_timeline' % self.base_url if user: url = '%s/%s.json' % (url, user) else: url = '%s.json' % url parameters = {} if count is not None: try: if int(count) > 100: raise TwitterError("'count' may not be greater than 100") except ValueError: raise TwitterError("'count' must be an integer") parameters['count'] = count if page is not None: try: parameters['page'] = int(page) except ValueError: raise TwitterError("'page' must be an integer") if since_id: parameters['since_id'] = since_id if retweets: parameters['include_rts'] = True if include_entities: parameters['include_entities'] = True json = self._FetchUrl(url, parameters=parameters) data = self._ParseAndCheckTwitter(json) return [Status.NewFromJsonDict(x) for x in data] def GetUserTimeline(self, id=None, user_id=None, screen_name=None, since_id=None, max_id=None, count=None, page=None, include_rts=None, include_entities=None): '''Fetch the sequence of public Status messages for a single user. The twitter.Api instance must be authenticated if the user is private. Args: id: Specifies the ID or screen name of the user for whom to return the user_timeline. [Optional] user_id: Specfies the ID of the user for whom to return the user_timeline. Helpful for disambiguating when a valid user ID is also a valid screen name. [Optional] screen_name: Specfies the scre

Source code and errata for "The Python Standard Library By Example"

Python is an object oriented high level programming language. One of the simplest programming languages of all, and also the most used language for creating System Security programs. It is Simple, yet the most powerful programming language which is very close to the Machine Language. If you have never programmed before, but you have little knowledge of how a computer works, then this the right place to start. After learning this course, one can start to build their own System programs, and also basic malware testing programs. This course is for those who want to learn how to program in python. It is intended to suit a wide audience, though not for absolute beginners. This course is targeted towards people who already have basic in either python or any other programming language. This course takes in consideration that you already know what loops, conditions, statements and variables are. This course will take you through the following chapters: Abstraction I and II, Exceptions in programs, Methods, properties and iterators, Standard Libraries and Python programming with Files. This will complete your intermediate training in Python Programming and how data structures are created and implemented in software’s. If you already know python, but other versions earlier than Python3.0, then you can start with this course to upgrade yourself with necessary modules. Some of the material in this course may be a bit difficult for an inexperienced programmer. However, once you start writing and practicing the examples in this course and writing your own codes based on these examples, then it will become quite practical for you to understand these terms. If you don’t understand some specific terms in the first bit, go through the tutorials again. Practice and Repetition are the keys to learning. This Course is in depth summary of the Core of Python. So, if you already have the basics in python, then it will be extremely easy to understand. Even if you don’t have the basics, then you can just search the specific terms used in here, understand only those and then come back to clear the advanced topics in these tutorials. This course will take you through the journey of interacting with system files using IDLE or the Python Interpreter, Creating your own servers and clients using socket programming, Understanding the database support for python with SQlite. There are also chapters which will teach you how to create your own programs which can interact with open files in any platform (UNIX, Linux, Windows or Mac). There is also a vast area covered in this course about Web Programming with Python covering the four most important topics as CGI, Screen Scraping, mod_python and Web Services. And finally there is testing, debugging and logging in Python. So, by completing this course, you will finally be able to write your own programs and test them on various platforms. The Main Idea of this course is to make the learner an advanced programmer having knowledge in bits and bytes in all various environments in Python programming. What are the requirements? Basics in Any Programming Language Basic Computer Knowledge Basics in Python such as ‘loops, variables, conditions and statements’(knowing any other language would be an added advantage) Basics in Python (knowing basic C,C++ and a bit about network programming would be an added advantage) What am I going to get from this course? Over 73 lectures and 10 hours of content! This course will clear advanced programming knowledge in Python Learn Abstraction I and II, Exceptions in programs, Methods, properties and iterators, Standard Libraries and Python programming with Files This course will take you through the journey of interacting with system files using IDLE or the Python Interpreter, Creating your own servers and clients using socket programming, Understanding the database support for python with SQlite. Learn Web Programming with Python covering the four most important topics as CGI, Screen Scraping, mod_python and Web Services Learn testing, debugging and logging in Python What is the target audience? Students Professionals Anyone who wants to learn a new Version i.e. Python 3.4.2

Course: CSE425 - Concepts of Programming Language Instructor’s Name: Adjunct Associate Professor Kamruddin Nur Section: 04 Assignment: Implement searching on the given data (NCHS_-_Leading_Causes_of_Death__United_States) in Comma Separated File Format Language Used: Shell script(mandatory), C plus plus(optional), Python(optional) Submission Deadline: 15th December, 2019 Student’s Name: Md. Mahmudul Haque Id No. : 1511944642 The code of the assignment that I have implemented are on three languages and before hopping into the contrast and comparison of those languages I am here giving a brief description on my procedure of coding. I used string searching Users are offered to search on their preference of column value While a match is found the program will return the entire row of that file I used for and while loop to search for the specific string that user might look for Exceptions are handled in C plus plus and Python whereas I used if else statement to handle exception in Shell Script Aliasing are used Type casting have been used in Python Concatenation of strings are used in all the languages. Now let’s see what are the comparison and contrast between the codes of those languages! Shell script: Key Points of my Code Description #no libraries needed to import! And no curly braces In shell script no libraries are needed to import explicitly. So it is indicating that this programming language is more easy to write and read for the non-programmers also. And for the curly braces, to implement the code, prior knowledge is required. while: #no condition More liberal while handling loop. Needs user interruption or programmer’s sentinels to stop infinity cycle. If else fi, case….esac, do… done To indicate the end of if-else block I have to write ‘fi’ term after each if-else block and similar happened to case … esac term under the case block and do … done block. Not readable and to write, one has to have prior knowledge. No need to declare the type of variables In Shell Script the type of variables are not needed to be declared by the programmer explicitly. So it is loosely typed and checking runtime. And least reliable. ‘ ;; ’ after every case block To mark the end of each case Shell script needs to be said the ‘ ;; ’ . * state in case block This is indicating the general block of case if the options does not match then this block will be operated. Not so readable. foo = foo + ‘,’ Or foo = ‘,’ + foo + ‘,’ Easy to concatenate, so easy to write. Also orthogonal. No exception handling I used if else fi instead. C plus plus(C++): Key Points of my Code Description #include <bits/stdc++.h> Libraries are needed to be declared explicitly. Different than Shell Script and Python. using namespace std It is used to indicate the compiler that the standard naming convention of c++ is being used. So it is more strict in terms of language building block. Different than Shell Script and Python. int main() { …. } To execute the whole code the code must be in between the curly braces of main function block. So this is supporting abstraction. Also the statements are compound and understandable if C is known. Different than Shell Script and Python. Type checking; example - string year; Usually check the type during compile time and strongly typed. Different than Shell Script. But python is loosely typed. while(1){ .. } To operate the while loop, the coder must at least need to assign it to a true value. Expressive and writeable though always need to have some knowledge on c++ before writing code using it. Also compound. Different than Shell Script and Python. ifstream Working as a input stream to read the file. Type and work specified. Similar to Python and Shell Script try{ …if(inFile) else throw “....” } catch(const char* e){ … } To handle exception, c++ has some built in library to handle it and user either can define or not to about the variation of exception he/she wants to handle. The try block is used as experiment part whereas the catch block will catch if there is no such file according to my code. Compound with form and meaning. Almost similar to Python but Shell script does not have this facility. bool flag This is indicating that c++ is capable of using concised parameters in terms of programming paradigm. But not orthogonal as I have to handle boolean value all the time. Similar to Python in some case but Shell Script does not have it. cout or cin Capable of printing and taking input through object as c++ allows object oriented programming whereas Python also support this. Python: Key Points of my Code Description flag = True Boolean data type exists and so it is concise which is similar to C++ but Shell Script does not have this print() Capable of procedural, functional, and object oriented job. C++ is almost similar but Shell script is totally different here. try: ….. except Exception: ……. To handle exception, python has some built in library to handle it and user either can define or not to about the variation of exception he/she wants to handle. The try block is used as experiment part whereas the catch block will catch if there is no such file according to my code. Compound with form and meaning like C++. file = open("NCHS_-_Leading_Causes_of_Death__United_States.csv") Python allows aliasing whereas C++ uses referencing in alias. with file: ... Helpful while handling files. Do not need to reopen and close the file every time. C++ also have similar to this but in stream format. But Shell script is more better in handling files option = input() option = int(option) Allows type checking like C++ but loosely typed like Shell script if... elif... Have conditional statements which are also in C++ and Shell Script Conclusion: So far according to my knowledge, I found out that while interacting with file Shell script is much more easier but also Python gives a good file handling libraries now like: Pandas and C++ is more prone to let user follow the programming paradigm while handling files.

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Guide to Web Scraping\n", "\n", "Let's get you started with web scraping and Python. Before we begin, here are some important rules to follow and understand:\n", "\n", "1. Always be respectful and try to get premission to scrape, do not bombard a website with scraping requests, otherwise your IP address may be blocked!\n", "2. Be aware that websites change often, meaning your code could go from working to totally broken from one day to the next.\n", "3. Pretty much every web scraping project of interest is a unique and custom job, so try your best to generalize the skills learned here.\n", "\n", "OK, let's get started with the basics!\n", "\n", "## Basic components of a WebSite\n", "\n", "### HTML\n", "HTML stands for Hypertext Markup Language and every website on the internet uses it to display information. Even the jupyter notebook system uses it to display this information in your browser. If you right click on a website and select \"View Page Source\" you can see the raw HTML of a web page. This is the information that Python will be looking at to grab information from. Let's take a look at a simple webpage's HTML:\n", "\n", " <!DOCTYPE html> \n", " <html> \n", " <head>\n", " <title>Title on Browser Tab</title>\n", " </head>\n", " <body>\n", " <h1> Website Header </h1>\n", " <p> Some Paragraph </p>\n", " <body>\n", " </html>" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's breakdown these components.\n", "\n", "Every <tag> indicates a specific block type on the webpage:\n", "\n", " 1.<DOCTYPE html> HTML documents will always start with this type declaration, letting the browser know its an HTML file.\n", " 2. The component blocks of the HTML document are placed between <html> and </html>.\n", " 3. Meta data and script connections (like a link to a CSS file or a JS file) are often placed in the <head> block.\n", " 4. The <title> tag block defines the title of the webpage (its what shows up in the tab of a website you're visiting).\n", " 5. Is between <body> and </body> tags are the blocks that will be visible to the site visitor.\n", " 6. Headings are defined by the <h1> through <h6> tags, where the number represents the size of the heading.\n", " 7. Paragraphs are defined by the <p> tag, this is essentially just normal text on the website.\n", "\n", " There are many more tags than just these, such as <a> for hyperlinks, <table> for tables, <tr> for table rows, and <td> for table columns, and more!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CSS\n", "\n", "CSS stands for Cascading Style Sheets, this is what gives \"style\" to a website, including colors and fonts, and even some animations! CSS uses tags such as **id** or **class** to connect an HTML element to a CSS feature, such as a particular color. **id** is a unique id for an HTML tag and must be unique within the HTML document, basically a single use connection. **class** defines a general style that can then be linked to multiple HTML tags. Basically if you only want a single html tag to be red, you would use an id tag, if you wanted several HTML tags/blocks to be red, you would create a class in your CSS doc and then link it to the rest of these blocks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scraping Guidelines\n", "\n", "Keep in mind you should always have permission for the website you are scraping! Check a websites terms and conditions for more info. Also keep in mind that a computer can send requests to a website very fast, so a website may block your computer's ip address if you send too many requests too quickly. Lastly, websites change all the time! You will most likely need to update your code often for long term web-scraping jobs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Web Scraping with Python\n", "\n", "There are a few libraries you will need, you can go to your command line and install them with conda install (if you are using anaconda distribution), or pip install for other python distributions.\n", "\n", " conda install requests\n", " conda install lxml\n", " conda install bs4\n", " \n", "if you are not using the Anaconda Installation, you can use **pip install** instead of **conda install**, for example:\n", "\n", " pip install requests\n", " pip install lxml\n", " pip install bs4\n", " \n", "Now let's see what we can do with these libraries.\n", "\n", "----" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example Task 0 - Grabbing the title of a page\n", "\n", "Let's start very simple, we will grab the title of a page. Remember that this is the HTML block with the **title** tag. For this task we will use **www.example.com** which is a website specifically made to serve as an example domain. Let's go through the main steps:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import requests" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Step 1: Use the requests library to grab the page\n", "# Note, this may fail if you have a firewall blocking Python/Jupyter \n", "# Note sometimes you need to run this twice if it fails the first time\n", "res = requests.get(\"http://www.example.com\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This object is a requests.models.Response object and it actually contains the information from the website, for example:" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "requests.models.Response" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(res)" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'<!doctype html>\\n<html>\\n<head>\\n <title>Example Domain</title>\\n\\n <meta charset=\"utf-8\" />\\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\\n <style type=\"text/css\">\\n body {\\n background-color: #f0f0f2;\\n margin: 0;\\n padding: 0;\\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\\n \\n }\\n div {\\n width: 600px;\\n margin: 5em auto;\\n padding: 2em;\\n background-color: #fdfdff;\\n border-radius: 0.5em;\\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\\n }\\n a:link, a:visited {\\n color: #38488f;\\n text-decoration: none;\\n }\\n @media (max-width: 700px) {\\n div {\\n margin: 0 auto;\\n width: auto;\\n }\\n }\\n </style> \\n</head>\\n\\n<body>\\n<div>\\n <h1>Example Domain</h1>\\n <p>This domain is for use in illustrative examples in documents. You may use this\\n domain in literature without prior coordination or asking for permission.</p>\\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\\n</div>\\n</body>\\n</html>\\n'" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res.text" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "____\n", "Now we use BeautifulSoup to analyze the extracted page. Technically we could use our own custom script to loook for items in the string of **res.text** but the BeautifulSoup library already has lots of built-in tools and methods to grab information from a string of this nature (basically an HTML file). Using BeautifulSoup we can create a \"soup\" object that contains all the \"ingredients\" of the webpage. Don't ask me about the weird library names, I didn't choose them! :)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import bs4" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": true }, "outputs": [], "source": [ "soup = bs4.BeautifulSoup(res.text,\"lxml\")" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<!DOCTYPE html>\n", "<html>\n", "<head>\n", "<title>Example Domain</title>\n", "<meta charset=\"utf-8\"/>\n", "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-type\"/>\n", "<meta content=\"width=device-width, initial-scale=1\" name=\"viewport\"/>\n", "<style type=\"text/css\">\n", " body {\n", " background-color: #f0f0f2;\n", " margin: 0;\n", " padding: 0;\n", " font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n", " \n", " }\n", " div {\n", " width: 600px;\n", " margin: 5em auto;\n", " padding: 2em;\n", " background-color: #fdfdff;\n", " border-radius: 0.5em;\n", " box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n", " }\n", " a:link, a:visited {\n", " color: #38488f;\n", " text-decoration: none;\n", " }\n", " @media (max-width: 700px) {\n", " div {\n", " margin: 0 auto;\n", " width: auto;\n", " }\n", " }\n", " </style>\n", "</head>\n", "<body>\n", "<div>\n", "<h1>Example Domain</h1>\n", "<p>This domain is for use in illustrative examples in documents. You may use this\n", " domain in literature without prior coordination or asking for permission.</p>\n", "<p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n", "</div>\n", "</body>\n", "</html>" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's use the **.select()** method to grab elements. We are looking for the 'title' tag, so we will pass in 'title'\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[<title>Example Domain</title>]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soup.select('title')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice what is returned here, its actually a list containing all the title elements (along with their tags). You can use indexing or even looping to grab the elements from the list. Since this object it still a specialized tag, we cna use method calls to grab just the text." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "title_tag = soup.select('title')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "<title>Example Domain</title>" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "title_tag[0]" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "bs4.element.Tag" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(title_tag[0])" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Example Domain'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "title_tag[0].getText()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example Task 1 - Grabbing all elements of a class\n", "\n", "Let's try to grab all the section headings of the Wikipedia Article on Grace Hopper from this URL: https://en.wikipedia.org/wiki/Grace_Hopper" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# First get the request\n", "res = requests.get('https://en.wikipedia.org/wiki/Grace_Hopper')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Create a soup from request\n", "soup = bs4.BeautifulSoup(res.text,\"lxml\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now its time to figure out what we are actually looking for. Inspect the element on the page to see that the section headers have the class \"mw-headline\". Because this is a class and not a straight tag, we need to adhere to some syntax for CSS. In this case" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<table>\n", "\n", "<thead >\n", "<tr>\n", "<th>\n", "<p>Syntax to pass to the .select() method</p>\n", "</th>\n", "<th>\n", "<p>Match Results</p>\n", "</th>\n", "</tr>\n", "</thead>\n", "<tbody>\n", "<tr>\n", "<td>\n", "<p><code>soup.select('div')</code></p>\n", "</td>\n", "<td>\n", "<p>All elements with the <code>&lt;div&gt;</code> tag</p>\n", "</td>\n", "</tr>\n", "<tr>\n", "<td>\n", "<p><code>soup.select('#some_id')</code></p>\n", "</td>\n", "<td>\n", "<p>The HTML element containing the <code>id</code> attribute of <code>some_id</code></p>\n", "</td>\n", "</tr>\n", "<tr>\n", "<td>\n", "<p><code>soup.select('.notice')</code></p>\n", "</td>\n", "<td>\n", "<p>All the HTML elements with the CSS <code>class</code> named <code>notice</code></p>\n", "</td>\n", "</tr>\n", "<tr>\n", "<td>\n", "<p><code>soup.select('div span')</code></p>\n", "</td>\n", "<td>\n", "<p>Any elements named <code>&lt;span&gt;</code> that are within an element named <code>&lt;div&gt;</code></p>\n", "</td>\n", "</tr>\n", "<tr>\n", "<td>\n", "<p><code>soup.select('div &gt; span')</code></p>\n", "</td>\n", "<td>\n", "<p>Any elements named <code class=\"literal2\">&lt;span&gt;</code> that are <span><em >directly</em></span> within an element named <code class=\"literal2\">&lt;div&gt;</code>, with no other element in between</p>\n", "</td>\n", "</tr>\n", "<tr>\n", "\n", "</tr>\n", "</tbody>\n", "</table>" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[<span class=\"mw-headline\" id=\"Early_life_and_education\">Early life and education</span>,\n", " <span class=\"mw-headline\" id=\"Career\">Career</span>,\n", " <span class=\"mw-headline\" id=\"World_War_II\">World War II</span>,\n", " <span class=\"mw-headline\" id=\"UNIVAC\">UNIVAC</span>,\n", " <span class=\"mw-headline\" id=\"COBOL\">COBOL</span>,\n", " <span class=\"mw-headline\" id=\"Standards\">Standards</span>,\n", " <span class=\"mw-headline\" id=\"Retirement\">Retirement</span>,\n", " <span class=\"mw-headline\" id=\"Post-retirement\">Post-retirement</span>,\n", " <span class=\"mw-headline\" id=\"Anecdotes\">Anecdotes</span>,\n", " <span class=\"mw-headline\" id=\"Death\">Death</span>,\n", " <span class=\"mw-headline\" id=\"Dates_of_rank\">Dates of rank</span>,\n", " <span class=\"mw-headline\" id=\"Awards_and_honors\">Awards and honors</span>,\n", " <span class=\"mw-headline\" id=\"Military_awards\">Military awards</span>,\n", " <span class=\"mw-headline\" id=\"Other_awards\">Other awards</span>,\n", " <span class=\"mw-headline\" id=\"Legacy\">Legacy</span>,\n", " <span class=\"mw-headline\" id=\"Places\">Places</span>,\n", " <span class=\"mw-headline\" id=\"Programs\">Programs</span>,\n", " <span class=\"mw-headline\" id=\"In_popular_culture\">In popular culture</span>,\n", " <span class=\"mw-headline\" id=\"Grace_Hopper_Celebration_of_Women_in_Computing\">Grace Hopper Celebration of Women in Computing</span>,\n", " <span class=\"mw-headline\" id=\"Notes\">Notes</span>,\n", " <span class=\"mw-headline\" id=\"Obituary_notices\">Obituary notices</span>,\n", " <span class=\"mw-headline\" id=\"See_also\">See also</span>,\n", " <span class=\"mw-headline\" id=\"References\">References</span>,\n", " <span class=\"mw-headline\" id=\"Further_reading\">Further reading</span>,\n", " <span class=\"mw-headline\" id=\"External_links\">External links</span>]" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# note depending on your IP Address, \n", "# this class may be called something different\n", "soup.select(\".toctext\")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Early life and education\n", "Career\n", "World War II\n", "UNIVAC\n", "COBOL\n", "Standards\n", "Retirement\n", "Post-retirement\n", "Anecdotes\n", "Death\n", "Dates of rank\n", "Awards and honors\n", "Military awards\n", "Other awards\n", "Legacy\n", "Places\n", "Programs\n", "In popular culture\n", "Grace Hopper Celebration of Women in Computing\n", "Notes\n", "Obituary notices\n", "See also\n", "References\n", "Further reading\n", "External links\n" ] } ], "source": [ "for item in soup.select(\".toctext\"):\n", " print(item.text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example Task 3 - Getting an Image from a Website\n", "\n", "Let's attempt to grab the image of the Deep Blue Computer from this wikipedia article: https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": true }, "outputs": [], "source": [ "res = requests.get(\"https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)\")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "soup = bs4.BeautifulSoup(res.text,'lxml')" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "image_info = soup.select('.thumbimage')" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[<img alt=\"\" class=\"thumbimage\" data-file-height=\"601\" data-file-width=\"400\" decoding=\"async\" height=\"331\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/b/be/Deep_Blue.jpg/220px-Deep_Blue.jpg\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/b/be/Deep_Blue.jpg/330px-Deep_Blue.jpg 1.5x, //upload.wikimedia.org/wikipedia/commons/b/be/Deep_Blue.jpg 2x\" width=\"220\"/>,\n", " <img alt=\"\" class=\"thumbimage\" data-file-height=\"600\" data-file-width=\"800\" decoding=\"async\" height=\"165\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Kasparov_Magath_1985_Hamburg-2.png/220px-Kasparov_Magath_1985_Hamburg-2.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Kasparov_Magath_1985_Hamburg-2.png/330px-Kasparov_Magath_1985_Hamburg-2.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Kasparov_Magath_1985_Hamburg-2.png/440px-Kasparov_Magath_1985_Hamburg-2.png 2x\" width=\"220\"/>]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "image_info" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(image_info)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": true }, "outputs": [], "source": [ "computer = image_info[0]" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "bs4.element.Tag" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ],

# to create an IMAP4 instance, preferably the SSL variant for security, connected to the Gmail server at imap.gmail.com: #If the login is successful, we can now do IMAPy things with our IMAP4 object. Most methods of IMAP4 return a tuple where the first element is the return status of the operation (usually 'OK' for success), and the second element will be either a string or tuple with data from the operation. #For example, to get a list of mailboxes on the server, we can call list(): #With Gmail, this will return a list of labels. To open one of the mailboxes/labels, call select(): #So with the mailbox selected, we can now get the emails within it. For example, we can get all the emails in the selected mailbox and for each one output the message number, subject, and date: #We use the search() method to get a list of message sequence numbers, then loop over these, calling fetch() to get the actual messages. #fetch() returns the raw message contents. To avoid having to parse the actual message data from fetch() ourselves, we can use the email package from the standard library. Once again, there are a few different packages floating around for doing this kind of thing, but I think email is currently the one least likely to get you down-voted on Stack Overflow. #message_from_string() returns a message object, and we can then access header items as a dictionary on that object. #Which brings us to the “Date” header, and the potentially thorny issue of date and timezone. If you don’t care about the date/time the emails were sent, then things are much simpler. But if you do care about such matters, note that the contents of the “Date” header may vary depending on the email client sending the email, and the timezone of the sender. Reliably converting to local time can be surprisingly tricky. Python is once again comes to the rescue with numerous modules promising to assist you with all your deepest darkest date conversion needs. The code snippet above shows one possible way of converting to local time, using the capabilities of email.util. It’s currently working well enough for my purposes, but there may be better ways to accomplish this. #The message body can be obtained by calling msg.get_payload(), which will return the payload data as a string (if the message is not multi-part). For text messages, you could then parse the data using regular expressions. For parsing contents of HTML emails however, you must not use regular expressions. Ever. Or you will feel the Lovecraftian wrath of the Stack Overflow minions. Instead, use a HTML parser, or higher level scraper like Beautiful Soup.

Framework improvement strategies and structures will quite often give an exhaustive and ideal answer for a given genuine issue. Traditional models, for example, SDLC and its stages give a typical guide to handle confounded issues in our reality. Be that as it may, such strategies don't take into contemplations the complex secret points of reality and client's requests. All in all the most accentuation is on the predefined and clear cut "request" 's and "struggle" 's are not tended to efficiently. Especially, in web application improvement, utilizing just specific objective standards isn't adequate in giving an exhaustive rundown of the necessities of the application. All in all, "emotional" factors should be tended to at the same time related to genuine ones and inside the system of request/struggle acknowledgment. This places of business web application improvement according to an alternate point of view. The "Do"s and "Do not"s are acknowledged along with emotional goal factors in a brought together system for reasonable and simple to-do web application improvement. Book description This book begins with a prologue to the web, including a short history of the TCT/IP convention and overall web. It characterizes the fundamental ideas for web servers and studies the instance of Apache, the most utilized webserver, while other free programming webservers are not neglected. The course go on with page plan focusing on HTML and JavaScript. XML Compositions, their approval and change are covered as well as powerful site pages worked with CGI, PHP or JSP and data set admittance. web services are programming parts that are open through Cleanser and have their connection point portrayed with WSDL (Web Administration Depiction Language). In this part the XML-RPC convention is talked about in addition to other things. All in all the most accentuation is on the predefined and clear cut "request" 's and "struggle" 's are not tended to efficiently in web applications. The last piece of the course manages setup, support, checking and security angles. About the Creators N/A Audits, Appraisals, and Suggestions: Amazon Related Book Classifications: Web Programming and Improvement Web Administrations Computer programming Peruse and Download Connections: Prologue to Web Applications Advancement (Carles Mateu) The Mirror Site (1) - PDF Comparative Books: O'Reilly® Endeavor Web Improvement: From Work area to Portable Creating Single Page Web Applications with Backbone.js (Prateek Dayal) Expert Web 2.0 Mashups: Remixing Information and Web Administrations (R. Yee) Basically, Web Applications are dynamic sites joined with server side programming which give functionalities, for example, interfacing with clients, associating with back-end data sets, and creating results to programs. Instances of Web Applications are Web based Banking, Long range interpersonal communication, Online Reservations, Internet business/ Shopping basket Applications, Intuitive Games, Internet Preparing, Online Surveys, Sites, Online Gatherings, Content The executives Frameworks, and so forth.. Advancements There are two fundamental classifications of coding, prearranging and programming for making Web Applications: I. Client Side Prearranging/Coding - Client Side Prearranging is the sort of code that is executed or deciphered by programs. web applications Client Side Prearranging is for the most part distinguishable by any guest to a site in web applications (from the view menu click on "View Source" to see the source code). The following are some normal Client Side Prearranging advancements:  HTML (HyperText Markup Language)  CSS (Flowing Templates)  JavaScript  Ajax (Nonconcurrent JavaScript and XML)  jQuery (JavaScript Structure Library - regularly utilized in Ajax improvement)  MooTools (JavaScript System Library - ordinarily utilized in Ajax improvement)  Dojo Toolbox (JavaScript Structure Library - normally utilized in Ajax advancement) II. Server Side Prearranging/Coding - Server Side Prearranging is the kind of code that is executed or deciphered by the web server. Server Side Prearranging isn't perceptible or available by any guest or overall population. The following are the normal Server Side Prearranging advancements:  PHP (exceptionally normal Server Side Prearranging language - Linux/Unix based Open Source - free reallocation, ordinarily consolidates with MySQL information base)  Zend Structure (PHP's Article Arranged Web Application System)  ASP (Microsoft Web Server (IIS) Prearranging language)  ASP.NET (Microsoft's Internet Application System - replacement of ASP)  ColdFusion (Adobe's Internet Application Structure)  Ruby on Rails (Ruby programming's Internet Application Structure - free rearrangement)  Perl (universally useful undeniable level programming language and Server Side Prearranging Language - free reallocation - lost its fame to PHP)  Python (universally useful significant level programming language and Server Side Prearranging language - free reallocation)

No description available

The code from the book "The Python Standard Library By Example"

Jupyter notebook for book 《The python 2 standard library by example》

《Python3标准库》书籍

《Python3标准库》

No description available

The Python Standard Library by Example

《The Python Standard Library by Example》，Take notes by python notebook.

the Python Standard Library by Example 一书的例子

Slides and Examples from the 3rd chapter of the Python Standard Library by Example book

The Python Standard Library by Example in ipynb. Examples are from the same-name book published by Doug Hellmann

My theoretical hypothesis is; “The evaluations of citizens’ voting about foreign movies are not influenced by the governments' official policy”. Because of the exhaustive hypothesis, it is not easy to collect data. So I decided to restrict the hypothesis in the example of the US and its politically affair tense countries; Russia and Iran. I used “the ratings of US citizens about Iranian and Russian movies” and “the ratings of Non-US people about Iranian and Russian movies” data from IMDB. The data was scraped (collected) from imdb.com with Python BeautifulSoup library.Mean, standard deviation (std) and p-value of the data were calculated by Python NumPy library. Also, The Pearson's r-value is calculated for each (Iranian and Russian) dataset and visualizing the result by Python SeaBorn library.

This project is about registering the complaints, by giving the information of the person who is going to register a complaint .For example by providing the name, address , gender, description about the complaint and submitting there complaint. We can also view the previous complaints registered in a database file. We will create a GUI application , by using tkinter module , which is the standard GUI library for python , after entering the details by the user a file will be created and the data entered by the user will be stored in a database file. Modules Used: 1.Tkinter 2. Sqlite3 To register complaint, a person need to create a new account (Sign Up) or login to the existing account through their username and password.

# Python Tkinter Calculator A calculator developed in Python using the standard Tkinter library. This project focused on creating an interactive GUI, with key bindings, in order to achieve an interface that is user-friendly. ## Running the Program Within the calulcator, the user will be able to use keyboard or button inputs. When the the operation is submitted, Python will evaluate the string using the eval() method to determine the solution. The eval function allows for standard order of operations to be recognized. **Ex:** ``` string = "" string += "1+2" # Operation submitted by user print(eval(string)) # Will print an evaluated string ``` ## Functionality This calculator can support parentheses use and decimal numbers. It can support basic operations such as: * Addition * Subtraction * Multiplication * Division * Squareroot * Exponents However, this calculator does not allow for implicit multiplication. For example, if the user was to input: ``` (1+2)(2+1) # 3*3 = 9 # Correct: (1+2)*(2+1) ```

A first look at a neural network (Deep Learning With Python, Francois Chollet Coursework). Let’s look at a concrete example of a neural network that uses the Python library Keras to learn to classify handwritten digits. The problem we’re trying to solve here is to classify grayscale images of handwritten digits (28 × 28 pixels) into their 10 categories (0 through 9). We’ll use the MNIST dataset, a classic in the machine-learning community, which has been around almost as long as the field itself and has been intensively studied. It’s a set of 60,000 training images, plus 10,000 test images, assembled by the National Institute of Standards and Technology (the NIST in MNIST) in the 1980s.