Projects > Opengear Cellular Checks

Description

When scheduled via a cronjob this script will allow you to periodically check all of your Opengear devices over the mgmt interfaces. The script than runs a series of checks against the device to determine the status of the cellular modem. If an issue is detected the script will reset the cell modem and schedule a reboot of the device.

#Script created by Brian Winning Jr. 
from __future__ import print_function	#Imports print module from future, for python3 print syntax in pre3 versions.
import sys, paramiko	#Imports paramiko module that enables us to open SSH connections.
import smtplib	#Imports smtplib module that enables us to open an smtp connection and send mail.
#Device List, the script will connect to these devices.
opengears = ["device1.yourdomain.com","device2.yourdomain.com","device3.yourdomain.com","device4.yourdomain.com","device5.yourdomain.com","device6.yourdomain.com","device7.yourdomain.com","device8.yourdomain.com","device9.yourdomain.com","device10.yourdomain.com","device11.yourdomain.com"]
#Use test group to test changes by updating hostname under client.connect and for loop
testgroup = ["dev-device1.yourdomain.com", "dev-device2.yourdomain.com","dev-device3.yourdomain.com"]
print("Opengear Cellular Stability Program\n")	#Opening statement the user will see when this script is first run.
print("This program will run checks against all Opengear Units for faults in the cellular subsystem, if any are found remediation steps will be taken\n")	#A brief description about the script to the user
print('Please allow a few minutes while the following devices are checked:\n') #Further information for the user on what to expect while the script is running and what devices will be checked.
for opengear in opengears:	#For loop used to print list of all devices in the list
	print(opengear)	#Prints device named as listed in list opengears
print("")	#empty space before next line.
mailserver = 'smtp.yourdomain.com'	#The FQDN of your mail server.
mailport = 25	#The port you connect to your mail server on.
mailsender = '[email protected]' #The email address where you wish to have received emails come from.
mailrecipient = '[email protected]'	#The email address where you wish to receive alerts generated by the script.
clientport = 22		#The port that SSH is available on for the Opengear devices.
clienttimeout = 3	#This timeout determines how long the script waits for a device to respond before moving onto the next device in the list.
clientuser = 'root' #User needs to be root to access cellctl commands
clientpass = 'Your Password'	#Your Opengear root user password.
client = paramiko.SSHClient()	#Loads Paramiko
client.load_system_host_keys()	#loads system SSH host keys into scripts for use
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())	#Adds any new devices in the list or change of SSH host keys to the users hosts file if they are not already present. This is needed if your have not connected to the devices from the host where the script is being run before.
for opengear1 in opengears:	#For loop that is used to connect to all of the devices, this helps keep the script simple and short. All the below commands until the indentation ends will be repeated for each device in the list.
	try:	#starts capture for error handling.
		client.connect(hostname = opengear1, username = clientuser,password = clientpass, port = clientport, timeout = clienttimeout)	#makes the initial connection to the devices and specifies the parameters we set above in the script.
		stdin, stdout, stderr = client.exec_command('ifconfig | grep wwan')	#Executes the command in quotations on the device and writes to the variable stdout
		stdin1, stdout1, stderr1 = client.exec_command('route | grep default | grep wwan') #Executes the command in quotations on the device and writes to the variable stdout1
		output = stdout.readlines()		##Writes all lines in stdout variable to output
		output1 = stdout1.readlines()	#Writes all lines in stdout1 variable to output1
		print("#################################################################")	#Text seperator in the script for cosmetic reasons.
		print("" ,opengear1, "being checked")										#Prints Opengear devices name followed by being checked.
		print("#################################################################")	#Text seperator in the script for cosmetic reasons.
		print("ifconfig | grep wwan output being checked against:\n")		#Prints to screen the command being checked against for check1.			
		print (''.join(output)) #Joins the several strings of output captured in the output variable into one.
		print("route | grep default | grep wwan output being checked against:\n")	#Prints to screen the command being checked against for check2.
		print (''.join(output1))	#Joins the several strings of output captured in the output1 variable into one.
		print("Status Of Checks:")	#Prints to the screen
		print("--------------------------------------")	#Prints to the screen a divider line for cosmetic reasons
		if 'wwan' not in ''.join(output):	#If/then statement checking for the presence of wwan in joined output variable. A match is if output does not contain wwan.
			print('Check 1 Failed for' ,opengear1, ',wwan not present in ifconfig\n')	#Print to screen that check1 failed for device and explaining why it failed.
			print("Rebooting Cell Modem and Reloading Device in 60 seconds\n")	#Print to screen what action is being taken
			stdin2, stdout2, stderr2 = client.exec_command('cellctl -r')	#Executes command in quotations on the device to reset the cell modem.
			stdin3, stdout3, stderr3 = client.exec_command('reboot -d 60 &')	#Schedules reboot of the device in 60 seconds to give the cell modem proper time to come back up.
			output2 = stdout2.readlines()	#Writes all lines in stdout2 variable to output2
			output3 = stdout3.readlines()	#Writes all lines in stdout3 variable to output3
			print (''.join(output2))	#Prints the several strings of output captured in the output2 variable into one.
			print (''.join(output3))	#Prints the several strings of output captured in the output3 variable into one.
			msg = 'Subject: {}\n\n{}'.format (str(opengear1) + ' reset Action Taken', str(opengear1) + " has had its cell modem reset and device has been reloaded due to failure of check1: wwan not present in ifconfig\n Recommended Action: Validate Opengear is back up and running properly.") #Email sent if Check1 for the device has failed, Subject is inserted as header.
			server = smtplib.SMTP(mailserver, mailport) #Specifies what server to use for sending the email message.
			server.sendmail(mailsender, mailrecipient, msg) #Opens an smtp connection to the server and sends the message.
			server.quit()	#Closes the SMTP connection
		else:	#Specifies action to take if Check 1 does not fail.
			print("Check 1 Passed for" ,opengear1, ",wwan is present in ifconfig\n") #Prints to screen that Check 1 passed and decribes what Check 1 is to the user.
			print("No action will be taken\n")	#Prints to the screen that no action will be taken for Check 1 to further clarify Pass.
		if 'UG' not in ''.join(output1):	#If/Then statement checking for the presence of UG  in the joined output1 variable. If UG is not present it determines the device should be reset. UG is the route flags for UP and Gateway.
			print("Check 2 Failed for" ,opengear1, "wwan default gateway route is not present.\n") 
            print("Rebooting Cell Modem and Reloading Device in 60 seconds\n")	
			stdin4, stdout4, stderr4 = client.exec_command('cellctl -r')	#Executes command in quotations on the device to reset the cell modem.
            stdin5, stdout5, stderr5 = client.exec_command('reboot -d 60 &') #Schedules reboot of the device in 60 seconds to give the cell modem proper time to come back up.
			output4 = stdout4.readlines()	#Writes all lines in stdout4 variable to output4
            output5 = stdout5.readlines()	#Writes all lines in stdout5 variable to output5
			print (''.join(output4))
            print (''.join(output5))
			msg1 = 'Subject: {}\n\n{}'.format (str(opengear1) + ' reset Action Taken', str(opengear1) + " has had its cell modem reset and device has been reloaded due to failure of check2: Default Route for WWAN Not Found.\n Recommended Action: Validate Opengear is back up and running properly")	##Email sent if Check2 for the device has failed, Subject is inserted as header.
			server = smtplib.SMTP(mailserver, mailport)	#Specifies what server to use for sending the email message.
			server.sendmail(mailsender, mailrecipient, msg1)	#Opens an smtp connection to the server and sends the message.
			server.quit()	#Closes the SMTP connection
		else:	#Specifies action to take if Check 2 does not fail.
			print("Check 2 Passed for" ,opengear1, "wwan default gateway route is present\n")	#Prints to the screen that Check 2 Passed and what was being checked.
            print("No action will be taken\n")	#Prints that no action will be taken, further clarifying Pass of Check 2

	except paramiko.SSHException:	#Specifies Specific Error Condition
		print("Connection Failed")	#Prints to screen user-friendly error message
		pass 		#Take No Action
		continue 	#Specifies Loop Should Continue to Next List Item Despite Error
	except paramiko.SSHException, socket.timeout:	#Specifies Specific Error Condition
		print("Socket Timeout - Script Will Continue")	#Prints to screen user-friendly error message
		pass 		#Take No Action
		continue 	#Specifies Loop Should Continue to Next List Item Despite Error
	except paramiko.SSHException, socket.error:		#Specifies Specific Error Condition
		print("Socket Error - Script Will Continue")	#Prints to screen user-friendly error message
		pass 			#Take No Action
		continue 		##Specifies Loop Should Continue to Next List Item Despite Error
	except Exception:	#Specifies Error Condition
		print(str(opengear1) + " will be skipped undefined Error - Script Will Continue\n")	#Prints to screen user-friendly error message
		pass 		#Take No Action
		continue 	#Specifies Loop Should Continue to Next List Item Despite Error

for opengear2 in opengears:	#For loop that loops through the entire list of Opengear devices and issues a client.close(), to close out each device session.
	client.close()	#Closes SSH connection

Features

  • Checks if Cell Modem is Active and Corrects Action
  • Check1: Checks if WWAN interface is up, which is an interface created by the cell modem.
  • Check2: Checks if WWAN default gateway route is present in the routing table.
  • Tested on Opengear ACM7004 and ICM7200
  • Easy to Add Devices: Devices are easily added to device list without appending further code to the script.

Download




Download Verification:
  • SHA-512:bd968dd0b0b0724e2b0769ff603735d8814c63f3823d722e3b5eef0cebd47624953d2afafb4e8d0798139efca3f6e77be1e62882391c822bb9ffd3078e8eb8db
  • SHA-256:10d464d9fefe2158e68497861af10d00aec499fe16e53f615c30b84e16d88cc7
  • SHA-1:8ccfddee698fc9b673cba7f11d34dc6d9c8f3c31
  • MD5:c99eafd30438caefd2fd754286b9f9d6

  • Version 1.2: January 10th 2019. - Added support for dual-sim opengear models which may use the wwan0 or wwwan1 interface.
  • Version 1.1: First public release, October 26th 2018. - Added comments for easier reading.

Licensing

Copyright 2018 Brian Winning Jr

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.