Python for Unix and Linux System Administration by Noah Gift IPython manual (there are both HTML and PDF versions supplied with the. Contribute to shihyu/python_ebook development by creating an account on GitHub. Python for Unix and Linux System. Administration. Noah Gift andJeremy M. Jones . O'REILLY®. Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei •.
|Language:||English, Spanish, Arabic|
|Genre:||Fiction & Literature|
|Distribution:||Free* [*Registration Required]|
Mac OS X is built on top of UNIX. Linux is a variation of UNIX. Most modern Linux and UN A Python Book: Beginning Python, Advanced Python, and Python. Python for Unix and Linux System Administration [Noah Gift] (). Read more · Automating Linux and Unix System Administration, · Read more. Python is pre-installed on most Unix systems. • The pre-installed version may not be the most recent one ( and as of July '14).
In either case, creating documentation is often a critical aspect of your job. But if it is not a task that you find yourself longing to do, it might be rather neglected. Python can help here.
No, Python cannot write your documentation for you, but it can help you gather, format, and distribute the information to the intended parties. In this chapter, we are going to focus on: gathering, formatting, and distributing information about the programs you write.
The information that you are interested in sharing exists somewhere; it may be in a logfile somewhere; it may be in your head; it may be accessible as a result of some shell command that you execute; it may even be in a database somewhere.
The first thing you have to do is to gather that information. The next step in effectively sharing this information is to format the data in a way that makes it meaningful. Finally, you need to get this information to the people who are interested in it.
Is it most convenient for the interested parties to receive an email, or visit a website, or look at the files directly on a shared drive? Automated Information Gathering The first step of information sharing is gathering the information. Text processing contains examples of the ways to parse and extract various pieces of data from a larger body of text. One specific example in that chapter is parsing the client IP address, number of bytes transmitted, and HTTP status code out of each line in an Apache web server log.
Gathering information can be more involved than just locating and extracting certain pieces of data. Often, it can be a process that involves taking information from one format, such as an Apache logfile, and storing it in some intermediate format to be used at a later time. Examples of such data stores include relational databases, object databases, pickle files, CSV files, and plain-text files. The remainder of this section will attempt to bring together some of the concepts from the chapters on text processing and data persistence.
We will use the same library from the text processing. Here is a simple module that uses both the Apache log parsing module created in the previous chapter and the shelve module:! Shelve is a module from the Python Standard Library. We then open the Apache logfile, access. We iterate over each line in the logfile and use the Apache log parsing module to create a dictionary from each line.
We then add the number of bytes for this specific request to the total number of bytes already tallied in the shelve object for this client IP address. If there is no entry in the shelve object for this client IP address, the total is automatically set to zero. After iterating through all the lines in the logfile, we close the logfile and the shelve object.
Receiving Email You may not think of receiving email as a means of information gathering, but it really can be. Imagine that you have a number of servers, none of which can easily connect to the other, but each of which has email capabilities. If you have a script that monitors web applications on these servers by logging in and out every few minutes, you could use email as an information passing mechanism. POP3 is perhaps the more common of these two protocols, and accessing your email over POP3 using poplib is quite simple.
Retrieving email using POP3! Then, we connected to the mail server and gave it the defined username and password. Assuming that all is well and we actually have permission to look at the email for this account, we then iterate over the list of email files, retrieve them, and write them to a disk.
All it would take to delete the email is a call to dele after retr. Retrieving email using IMAP! Then we started iterating over a search of the entire directory. The search method is poorly documented in the Python Standard Library documentation. The two mandatory parameters for search are character set and search criterion. What is a valid character set?
What format should we put in there? What are the choices for search criteria? What format is required? For each iteration of the loop, we write the contents of the email to disk. By this, we mean information that you gather with your own eyes and key in with your own hands.
Examples include a list of servers with corresponding IP addresses and functions, a list of contacts with email addresses, phone numbers, and IM screen names, or the dates that members of your team are planning to be on vacation. There are certainly tools available that can manage most, if not, all of these types of information. There is Excel or OpenOffice Spreadsheet for managing the server list. There is Outlook or Address Book.
This may be the solution for the situations that arise when technologies are freely available and use an editing data format that is plain text and which provides output that is configurable and supports HTML or preferably XHTML. Here is how to call a Python script from a fancy Cocoa application: import "MyDocument.
Try inside the python framework if! Here is how the reStructuredText website describes it: reStructuredText is an easy-to-read, what-you-see-is-what-you-get plaintext markup syntax and parser system. It is useful for in-line program documentation such as Python docstrings , for quickly creating simple web pages, and for standalone documents.
The reStructuredText parser is a component of Docutils. ReST is the preferred format for Python documentation. If you create a Python package of your code and decide to upload it to the PyPI, reStructuredText is the expected documentation format. Many individual Python projects are also using ReST as the primary format for their documentation needs.
So why would you want to use ReST as a documentation format? First, because the format is uncomplicated. Second, there is an almost immediate familiarity with the markup. When you see the structure of a document, you quickly understand what the author intended. Now, we'll show a bulleted list: - item one - item two - item three That probably makes some sort of structured sense to you without having to read the documentation about what constitutes a valid reStructuredText file.
You might not be able to write a ReST text file, but you can probably follow along enough to read one. We imported docutils.
Then we defined a string that contained our reStructuredText, and ran the string through docutils. Every good sysadmin needs to keep track of the servers they have and the tasks those servers are being used for.
Or you can call the Textile conversion module from within some script and programmatically deal with the XHTML that is returned. Either way, the Textile markup and the Textile processing module can be hugely beneficial to your documenting needs.
For Ubuntu, the package name is python-textile, and you can install it with apt-get install python-textile. Once Textile is installed, you can start using it by simply importing it, creating a Textiler object, and calling a single method on that object.
There are plenty of resources on the Web for that. Function card If you do have data, such as server lists and contact lists, that needs to be converted into HTML and then have some action such as emailing the HTML to a list of recipients or FTPing the HTML to a web server somewhere taken upon it, then either the docutils or the Textile library could be a useful tool for you.
Information Formatting The next step in getting your information into the hands of your audience is formatting the data into a medium that is easily read and understood. We think of that medium as being something at least comprehensible to the user, but better yet, it can be something attractive. Graphical Images The following two examples will continue the example of parsing an Apache logfile for the client IP address and the number of bytes that were transferred.
In the previous section, our example generated a shelve file that contained some information that we want to share with other users. So, now, we will create a chart object from the shelve file to make the data easy to read:! Bar chart. We then opened the shelve file we created in the previous example. Since the shelve object shares the same interface as the builtin dictionary object, we were able to call the Items method on it.
One common, and horribly false, misconception is that only some people can learn to program, and they are a mysterious and elite few. The frank truth is that anyone can learn how to program. A second, equally false, misconception is that earning a computer science degree is the only way a person can truly become a software engineer.
But some of the most prolific software developers do not have engineering degrees. There are people with philosophy, journalism, nutritional science, and English degrees who are competent Python programmers. Another funny, and false, misconception is that you must have started to program in your teenage years, or you will never learn to program. While this makes people who were lucky enough to have someone in their life that encouraged them to program at a young age feel good, it is another myth.
It is very helpful to have started learning programming at a young age, but age is not a requirement to learn Python. There are countless cases of developers who learned to program in their late 20s, 30s, 40s, and onward. If you have gotten this far, we should point out that you, the reader, have an advantage many people do not.
If you decided to pick up a book on Python for Unix and Linux system administration, then you most likely know something about how to execute commands from a shell. This is a tremendous advantage to learning to become a Python programmer. Having an understanding of the way to execute commands from a terminal is all that is required for this introduction to Python. If you truly believe you will learn how to program with Python, then read the next section immediately.
It is really that simple; if you make this decision, it will change your life. You will need to open two terminal windows, one with IPython and one with Bash. In every example, we will compare what we do in Python with a Bash example. The first steps are to download the correct version of IPython for your platform and install it. You can also download a copy of the virtual machine that includes all of the software for the book, as we have a copy of IPython preconfigured and ready to run.
You just need to type in ipython, and you will get a prompt. Once you have installed IPython and have an IPython shell prompt, it should look something like this: [ngift Macintosh-7][H][J:0] ipython Python 2.
IPython 0. In : An IPython shell is quite a bit like a regular Bash shell and can execute commands such as ls, cd, and pwd, but you can read the next chapter for more of a scoop on IPython.
This chapter is about learning Python, so on to the tutorial. Executing Statements in Python If you spend a lot of your day typing commands into a terminal, then you are used to executing statements and, perhaps, redirecting the output to a file or to another Unix command.
In this particular situation, all that is important to know is that you import the subprocess and use it in the syntax that is shown. We will get into the particulars of how subprocess and import work later, but for now, ignore why it works and copy the code: subprocess. Given this bit of information, you can now create a Python version of ls. Just open up your favorite text editor in another terminal tab or window and place this in a file named pyls.
Python wrapper for ls command!
Given this new bit of information, you could happily start writing some basic scripts if you just put one command after another in a file and ran it. See Examples and System information script—Python! System information script—Bash! And if we run them, we see that the output of each is identical. One quick note though: splitting the command from the argument is completely optional using subprocess.
You can also use this syntax: subprocess. In the Python version of the script, we imported the subprocess module because it already contained the code to make system calls in Python. As we mentioned earlier, importing a module like subprocess is just importing a file that contains code you can use. You can create your own module or file and reuse code you have written in the same way you import subprocess. Importing is not magic at all, it is just a file with some code in it.
One of the nice things about the IPython shell that you have open is its ability to inspect inside modules and files, and see the attributes that are available inside them.
Modules such as subprocess are very similar. With IPython you can use tab complete to look at the tools that are available inside a module. Remember, a module is just a file with some code in it. Here is what a tab complete looks like with the subprocess module in IPython: In : subprocess. CalledProcessError subprocess. MAXFD subprocess. PIPE subprocess. Popen subprocess.
In the third column of our example, notice that you see subprocess. Now, to see more information about how to use subprocess. Wait for command to complete, then return the returncode attribute.
The arguments are the same as for the Popen constructor. In IPython, when you type a question mark after the attribute you want to find information about, the documentation that has been included with the attribute will print out. If you do this on most attributes in the standard library, you should find some helpful information to properly use them. Keep in mind that you can also refer to the Python Standard Library documentation as well.
Summary You now have enough information to call yourself a Python programmer. You know how to write a simple Python script, how to translate simple scripts from Bash and call them with Python, and, finally, how to find documentation about new modules and attributes.
Using Functions in Python In the previous section we went through executing statements one after another, which is pretty useful, because it means we were able to automate something that we would normally have to do manually.
The next step to automating our code execution is to create functions. If you are not already familiar with writing functions in Bash or another language, then one way to think about functions is as miniscripts. A function allows you to create blocks of statements that get called in groups that live inside of the function. This is quite a bit like the Bash script we wrote in which there were two commands enclosed in a script.
One of the differences between a Bash script and a function is that you can include many function scripts. Ultimately, you can have multiple functions that group statements together in a script, and then that group of statements can be called to run a miniprogram at the proper time in your script. At this point, we need to talk about the topic of whitespace. In Python, a uniform level of indentation must be maintained in nesting code.
In another language, like Bash, when you define a function you put brackets around the code inside of a function. With Python, you must indent the code inside of the bracket. This can trip up newcomers to the language, at first, but after a while it will grow on you, and you will realize that this encourages readable code. If you have trouble getting any of these examples to work interactively, make sure you refer to the actual source code to see the proper indentation level.
The most common practice is to set a tab to indent exactly four spaces. Just type the following into the interactive IPython prompt: In : def pyfunc In line , we use a programming idiom, or technique, that executes the function five times.
We can do the same thing in a live Bash shell as well. Here is one way: bash Notice the difference between the Bash for loop and the Python for loop. If this is your first exposure to a function in Bash or Python, you should make some other functions in your IPython window before you continue.
Functions are not magic, and writing multiple functions interactively is a great way to take away the mystery if this is your first experience with them. So we can go back to the script we wrote that prints system information and convert those statements into functions. If you are not familiar with this style, you might not have known that it is common to create several functions inside a script and then call them all with one main function.
One of many reasons for this is that if you decide to reuse this script for another program, you can either call the functions independently or together with the main method. The key is that you decide after the module is imported. When there is no control flow, or main function, then all of the code gets executed immediately when it is imported.
This may be OK for a one-off script, but if you plan to create reusable tools, and you should, then it is a good practice to create functions that encapsulate specific actions, and then have a main function that executes the whole program. We created two functions and then called those two functions by calling the main function. If this is your first experience with functions, then we would highly recommend that you comment out the main method by placing a pound sign in front of both the Bash and the Python scripts and running them again.
At this point, you are now a programmer capable of writing simple functions in both Bash and Python. Programmers learn by doing, though, so at this point we highly recommend that you change the system calls in these two Bash and Python programs and make them your own.
Give yourself some bonus points if you add several new functions to the script and call them from a main function. Reusing Code with the Import Statement One problem with learning something new is that, if it is abstract, like calculus, for example, it is hard to justify caring about it.
When was the last time you used the math you learned in high school at the grocery store? In our previous examples, we showed you how to create functions as an alternative to executing shell commands one after another in a script. We also told you that a module is really just a script, or some lines of code in a file. Here is the point where we show you why you should care. Open the IPython and Bash windows if you closed them so that we can demonstrate very quickly why functions are important for code reuse.
One of the first scripts we created in Python was a sequence of commands in a file named pysysinfo. In Python because a file is a module and vice versa, we can import this script file into IPython.
Keep in mind that you never need to specify the. In fact if you do this, the import will not work. If you import a file full of Python code it seems to runs great. But, actually, there are a few problems with this.