𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐞 𝐇𝐚𝐝𝐨𝐨𝐩, 𝐀𝐖𝐒 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐨𝐜𝐤𝐞𝐫, 𝐋𝐕𝐌 And 𝐖𝐞𝐛𝐬𝐞𝐫𝐯𝐞𝐫 𝐔𝐬𝐢𝐧𝐠 𝐏𝐲𝐭𝐡𝐨𝐧

What is Automation?

Automation is the technology by which a process or procedure is performed with minimal human assistance. The process of creating software and systems to replace repeatable processes and reduce manual intervention.

What is Hadoop?

Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data.

What is Docker?

Docker is a software platform for building applications based on containers — small and lightweight execution environments that make shared use of the operating system kernel but otherwise run in isolation from one another.

What is AWS?

Amazon Elastic Compute Cloud (EC2) is a part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), that allows users to rent virtual computers on which to run their own computer applications. EC2 encourages the scalable deployment of applications by providing a web service through which a user can boot an Amazon Machine Image (AMI) to configure a virtual machine(Instance). A user can create, launch, and terminate server-instances as needed, paying by the second for active servers. hence the term “elastic”.

Here is my Article on Automation using python. In This article, created a menu-driven program that automates Hadoop, Docker, some services of AWS Cloud, Linux partitions and configuring webserver and User does not need to learn any command.

Automation Menu start Here?

Code:

AWS Cloud Automation:

AWS KEY-PAIR:

it is a set of security credentials that you use to prove your identity when connecting to an instance

Creat Key

and you don't need to worry about the key because it will automatically be stored on your computer.

To describe the key pair:

TO delete Key :

code:

AWS_Security_Group:

create security Group:

Display all security Groups:

Delete Security Group:

Delete Security Group

Add Rule to security Group:

When we create a Security Group, by default no rules are attached

Delete Rule:

I have added 22 and 80 using the menu again in the py security group.

Security-Group Code:

AWS EC2 Instance

TO see all instances :

To launch instance

Stop the instance and start the instance

Terminate Instance:

AWS EC2 Code:

AWS EBS VOLUMES:

List volumes:

Create Volume

Attach Volume:

Modify-Volume:

Detach Volume:

Delete Volume :

AWS EBS Volume code

AWS S3:

create an s3 bucket:

Upload all files from the local directory to s3

Delete Bucket Object:

Delete Bucket:

AWS S3 CODE:

AWS CloudFront:

Hadoop Automation

Hadoop menu starts here first you have to choose where you want to work local operating system or remote

let's work on remote and also all features are working correctly on local as well as on remote.

as you can see I launched two instances on AWS let set up name node and data node.

here menu asks you that does your VM has password base authentication or key-based authentication.

added all information of operating system like user name, IP address, and key location

Configure NameNode

before moving ahead we have to configure the core-site file and hdfs-site file first in the local menu code

/BIGDATA/templates/core-site…

/BIGDATA/templates/hdfs-site…..

Hadoop was successfully installed.

configure name nod, Format, and start name node service

configure NameNode and start service

I am not showing other options here and they are working great.

Hadoop Code:

This code supports all option on the remote operating system having a key,password-based authentication, and local operating system

LVM AUTOMATION:

z

remaining options are also working great and GitHub Link at the end.

LVM CODE:

the main menu of LVM is def LVM() at the end of the below code.

Automation of Docker:

Docker Automation Code:

Automate Webserver

configure webserver on

• Docker container running on local/remote[having key-based authentication or password-based authentication]
• Webserver on local operating system/remte[having key-based/password-based authentication]

Webserver Automation Code:

GitHub Link:

https://github.com/rohitraut3366/Automate_Hadoop_AWS_Docker_Using_Python

All credit goes to Team member:

Ishika Mandloi - Sandip Foundations Sandip Institute of Technology & Research Centre…

Rohit Raut - Python - ARTH - The School of Technologies | LinkedIn

Thank you for reading!!!😇😇😇