Basics of DevOps : Part 5 (END)

In 1st Part of Basics of DevOps, we discussed the basics including definition and origin of the terminology. In 2nd Part of Basics of DevOps; misconceptions, principles, requirements, benefits were discussed. In 3rd Part of Basics of DevOps, We Discussed About DevOps Team Covering Team Structure, Project Management, Design and Quality Management. In 4th Part of Basics of DevOps, We Discussed About DevOps Processes Covering Processes, Design, Design, Development and Testing. In This 5th and Final Part of Basics of DevOps, We Will Cover the Platforms and Tools for DevOps and Draw Overall Conclusion on the Whole Series.

Basics of DevOps : Platforms and Tools for DevOps

As explained in the previous articles, fundamental goal of the DevOps movement to bridge the gap between development and operations. Processes should be executed together. But in terms of the tools used, there is a clear gap between development and operation in traditional IT operations. So the goal must be to close this gap and find a set of tools that both working groups can accept and use. However, some people prefer to think about tools rather than employees and processes when confronted with DevOps. Without understanding the fundamental idea of ​​DevOps to improve collaboration and share processes, the introduction of new tools will fail. Some employees will feel that “only new tools are introduced”. So the big goal must be to use new tools to save time, allowing employees to focus on more serious problems. If a tool promise to unite all areas of DevOps, it should not be considered further.

Planning

As mentioned in previous articles on this series, DevOps implements agile development methods. An important part of this is the planning of upcoming tasks in certain sections. Atlassian has created a solution in this area with the “Jira Software” product. JIRA Software, in its basic function, is a ticket system commonly used for software bug tracking. However, thanks to its advanced customization options, it can be used for many more applications. This makes it a great support in agile development. JIRA Software implements Scrum as an agile method of software development . You can create tasks within the tool, manage a sprint backlog, identify team engagement and speed, and generate team progress reports. Here, project management (“managers”) and development (“users”) work together. The project manager initiates projects and plans a new version of the software to be developed. Sprints are performed in certain sections. A common running time of a sprint is here a week. At the beginning of this week the sprint will be scheduled. During this planning process, assignments are assigned to the individual developers, which have to be done during the runtime of the sprint. At the end of a sprint an analysis of the past week will be carried out and new insights, problems and their solutions will be recorded. Once all planning points of the version have been processed, the new version goes into the release.

Another component of each planning phase is to exchange and record knowledge. Atlassian has also developed a suitable platform called Confluence for this requirement. It has been available on the market for a long time, providing an early example of a web-based wiki platform. It is already used by a large number of companies. Confluence can be used in the Atlassian cloud or on its own server. Confluence covers, among other things, the following use cases :

• Preparation of a technical documentation
• Construction of a knowledge database
• Building an internal wiki

Software development in particular can use this platform to document their daily work, to share acquired knowledge or to gain access to product requirements. These requirements are then in turn incorporated by other teams in Confluence. This establishes a first interface between several departments.

Development

Development is also facing new challenges in the course of implementing DevOps. So it has to increasingly deal with aspects of the operation. Setting up physical or virtual machines or securing systems are just a small selection of these new tasks. But here too, support can be provided with the right selection of tools.

For many applications, new virtual machines are always set up in software development. They simulate environments or requirements. A big disadvantage of this solution is the huge size of a virtual machine. This often moves in the single-digit GByte range. As a result, portability and the further development by means of version management are made enormously more difficult. Here a supportive tool is Docker. It is an open source project and licensed under the Apache License 2.0, Docker provides containers with its architecture. In these containers it is possible to run any application with all its dependencies. It thus provides a uniform development environment.

The host system can be a dedicated or virtual machine with a Linux operating system. The Docker Engine is then installed on the machine and provides the functionality to create, deliver and operate Docker containers. This model allows you to start multiple programs in different independent containers. This results in a better utilization of the resources of the actual host system. In addition, it prevents erroneous interactions that can arise in a parallel operation of the application on a system. These are managed on a platform such as the “Docker Hub”. We have detailed basic series on Docker.

In addition to a unified development environment, the source code should also be managed in a form open to other teams. Thus, in addition to software processing, the company can directly access the created source code and extend it. Atlassian provides this platform with their product “Bitbucket”. It’s a Git-based source code versioning engine. A great advantage of this tool is that it has a connection to the JIRA software. This makes it possible to link the created source code with the respective planned task.

The benefits of using centralized source code versioning are many. Thus, as a leading medium, it makes it superfluous to manually adjust the individual machines. In addition, a direct connection with TeamCity is feasible. TeamCity listens for changes in the git repository on the Bitbucket server. If it detects a change, the configured build process is started automatically. The TeamCity tool will be presented below in the section “Continuous Integration”.

Continuous Integration

TeamCity is a continuous integration (CI) tool from JetBrains. As mentioned earlier, Teamcity starts a configured build process after changing source control. This process implies a compilation of the source code, the execution of unit tests and integration tests. Finally, the compiled and successfully tested source code is automatically compiled. A final distribution of the compiled software into a test environment can also be realized via TeamCity.

Basically, the entire build process depends on which programming language the software is implemented in. TeamCity offers the support of all common languages. Among other things, Java, C# and Ruby belong to the given scope of functions. Based on this, many plugins for individual extension can be integrated. In this way a solution suitable for the respective requirements can be realized. Docker is already supported by an existing project. It is thus possible to continuously transfer the existing development environment to test environments. The functionality also includes a RESTful API, This provides endpoints for performing remote actions. All common platforms are also supported as source code administration, including the source code management Git mentioned in the context of this development. As already mentioned, special “build agents” are responsible for executing the build process. These can be installed on several or different computers and used independently. Run-time statistics balance the load among multiple build agents. In addition to the features already presented, TeamCity can also be integrated directly into multiple development environments.,

Software Distribution

At the end of TeamCity’s build process is the compiled source code, which also needs to be automatically distributed to each environment. This implements the fully automated process from development to operation of the software at the customer. A manual intervention of the development or the operation is no longer necessary.

The Octopus Deploy product enables automated software distribution. It communicates directly with TeamCity via an interface. Basically, Octopus Deploy is a user-friendly and automated software distribution server. It is optimized for .NET applications. One goal of this tool is to enable repeatable and trustworthy software distribution. Although it has been optimized for .NET, it also supports other technologies. The distribution of a NodeJS application is already officially described in the documentation of the product. The Octopus Deploy Server relies on already introduced tools.

Automated software distribution results in a more regular distribution of software products across different environments at a much lower cost. The time saved gives each developer the opportunity to focus more on the actual coding. The change is also noticeable for the customer. Updates are faster than previously possible. A continuous and short-term response to reported problems becomes feasible.

Operation

If the software is developed, created and distributed after completion of all previous process steps, reliable operation must be guaranteed. For this, the application and its environment should be continuously monitored. Continuous monitoring using tools such as Nagios or Munin provides a detailed insight into the status of the monitored system running the software. The data and metrics made available in this way can also be used by other areas outside the company. Development can use this data to give stability to its software and quality assurance is thus able to develop measures.

For continuous monitoring during operation, a large amount of information is collected depending on the size of the environment. If, in addition to the variety of information, several tools are used for different systems, the problem arises. Exactly at the overview uses a very interesting tool called BigPanda. BigPanda is a data science platform that enables IT teams to effectively solve problems even in large and dynamic environments. Scaling, speed and fragmentation are so advanced here that it is a great challenge for the operation to keep track of the current status of all systems.

Big Panda

BigPanda is in the middle of the tools used for continuous monitoring. In addition to solutions for monitoring (Nagios, Cloudwatch) can also be integrated tools from the field of planning (JIRA) or chat platforms for team collaboration (Slack). From the monitoring systems, BigPanda collects notifications and presents them in a clear and prefiltered form. This allows the operations team to respond quickly to alerts. The other integrated tools can, for example, be used to create a task in JIRA directly in response to a fault that has occurred or to inform the team directly about the problem via a chat platform. A permanent review of all systems in detail is no longer necessary with the use of BigPanda.

Continuous feedback

With the tools already presented it is now possible to realize a maximum automated process from the planning to the operation of the software. During each of these steps different information is produced. Information about the performance, stability or results of the tests. But not only through the internal processes data is collected. The end user also produces a large variety of data when interacting with the software.

But it is precisely this information of the end user can only be evaluated with the help of a suitable tool. Pendo is one of those tools. Pendo collects, analyzes, and presents all user behavior data in a concise and customizable form. It also offers questionnaires that can be seamlessly integrated into an application.

Overview of DevOps Tools

Overall, all the tools presented are selected so that they can seamlessly mesh. Confluence and JIRA facilitate the planning of software projects. Docker and Bitbucket are used in development. Teamcity ensures the continuous integration of the developed source code. With Octopus, the compiled source code is distributed to the respective servers. With the help of BigPanda, the company has a complete overview of the monitoring tools used. The infinite DevOps process is completed with the help of continuous feedback via Pendo. Finally, it should be noted that all tools can only work together if a suitable technology is used. A suitable technology is a web application based on NodeJS, as it is supported by all tools.

The Conclusion

DevOps is becoming more and more popularround the globe. More and more companies dare to tackle the project “DevOps”. Others, however, shy away from the fact that their own IT department needs to be completely reoriented. After all, the changes affect the three major design fields of the IT organization: governance, work processes and automation.

The basics for introducing DevOps into business are provided by sufficient, high-quality tools. The challenge for IT departments in this sense is on the organizational side. Here, the entire company must join in to make the change process as simple and fast as possible. DevOps is already used in many development teams for smaller projects or for problem solutions based on agile principles. It often happens that with smaller errors the IT enterprise approaches the developer and finds with this a fast solution. For larger projects, however, it requires an organizational framework and supporting tools.

From a security perspective, there are shared opinions for the DevOps topic. Critics see “bad risks” in ill-considered implementations that potentially threaten the entire business operations of the company. Proponents of DevOps, however, see the security benefits as improved control of software distribution, as well as greater traceability and increased transparency. Companies must take appropriate action in advance to maximize the benefits of DevOps without taking any risks. Upon introduction, an external security representative can be brought in, It takes expert safety and compliance assessments into account right from the start, eliminating security risks and subsequent compliance issues.

The need for DevOps is dictated by the speed of the market – software developers simply have to deliver faster to stay profitable today. So the market is IT needs to keep up this pace while ensuring operational stability. However, these demands on IT can only be realized if everyone involved in the value chain works together. Exactly these requirements are “DevOps”. Many startups rely on the method from the beginning. However, the larger, already established companies hesitate to make the decision and take the time to observe the first results of other companies. But the fact is: the basic conditions under which classical software engineering originated no longer exist in this form, yet the outdated principles based on it continue to dominate most IT organizations. IT must adapt to the changed conditions and DevOps is an important building block on the way.