Building Bots with Mechanize and Selenium


Picking the right tool

The Sociosploit team conducts much of its research into the exploitation of social media using custom built bots. On occasion, the team will use public APIs (Application Programming Interfaces), but more often than not, these do not provide the same level of exploitative capabilities that could be achieved through browser automation. So to achieve this end, the Sociosploit team primarily uses a combination of two different Python libraries for building web bots for research. Each of the libraries have their own advantages and disadvantages. These libraries include:

  1. Mechanize
    • Pros:
      • Very lightweight, portable, and requires minimal resources
      • Easy to initially configure and install
      • Cuts down on superfluous requests (due to absense of JavaScript)
    • Cons:
      • Does not handle JavaScript or client-side functionality
      • Troubleshooting is done exclusively in text
  2. Selenium
    • Pros:
      • Operations are executed in browser, making JavaScript rendering and manipulation easy
      • Visibility of browser simplifies troubleshooting
    • Cons:
      • Painful to initially build bot environment
      • Scripts are not portable, as they require installation of supporting web browser API drivers
      • More resource intensive (due to browser usage), and often executes superfluous requests (due to JavaScript)

To demonstrate the basic functionality of each, we will craft a quick LinkedIn login function using both Mechanize and Selenium.

Mechanize

Import libraries and configure browser object

First we need to import the mechanize library, instantiate the browser object, and then configure it.

# Import Mechanize
import mechanize

# Instantiate Browser
li_bot = mechanize.Browser()

# Configure Browser
li_bot.set_handle_robots(False)
li_bot.set_handle_refresh(False)
Browse to the website and examine

Next we instantiate an instance of the browser object, then browse to linkedin's homepage and examine the contents (to include forms, links, or source-code).

# Browse to LinkedIn
response = li_bot.open('https://www.linkedin.com/')

# Review available forms
for form in li_bot.forms():
    print form

# Review available links
for link in li_bot.links():
    print link

# Review source code
response.read()
Interact with Form

Below is a snippet from the response for enumerating the forms. It is apparent that this is the login form, as it has an available TextControl for session_key (i.e. username) and session_password (i.e. password).

<POST https://www.linkedin.com/uas/login-submit application/x-www-form-urlencoded
  <TextControl(session_key=)>
  <PasswordControl(session_password=)>
  <SubmitControl(<None>=Sign in) (readonly)>
  <HiddenControl(isJsEnabled=false) (readonly)>
  <HiddenControl(loginCsrfParam=a1badbb6-d6c8-4b33-847e-79f75d99f920) (readonly)>>

Unfortunately, the form does not have a name identifer, so we have to interact with it by using the index value.

# Select form to interact with
li_bot.form = list(li_bot.forms())[0]

# Supply username to appropriate form control
username = li_bot.form.find_control('session_key')
username.value = 'notarealuser@gmail.com'

# Supply password to appropriate form control
password = li_bot.form.find_control('session_password')
password.value = 'P@$$W0rd123!!'

# Submit form
response = li_bot.submit()
Test Login Success

Finally, after logging in, we can confirm that login was successful by examining the updated browser content. One easy way to do this on most sites is to review the links, to determine if there is any exclusively post-authentication functionality (such as a Log Out function, user network management, account management, etc.).

>>> for link in li_bot.links():
...     if link.text == 'My Network':
...             loggedin = True
...     else:
...             pass
...
>>> if loggedin:
...     print "Login was Successful"
...
Login was Successful

Selenium

Import libraries and configure browser object

First we need to import the selenium library and instantiate the browser object.

from selenium import webdriver
driver = webdriver.Firefox()
Browse to the website and examine

Next we browse to the site and examine it.

driver.get('https://www.linkedin.com')

Unlike with Mechanize, this will actually start a browser window, and we can examine the contents in the browser. In Firefox, you can right click on any element in the browser, then click "Inspect Element".

Header

This will bring up the element in Inspector. From here, you can right click on the element field and then select Copy > XPath. This XPath can then be used to interact with the element using the Selenium browser object.

Header

Interact with the login form

We can use these XPath's to supply the username and password to the appropriate fields, and then to click the Submit button.

element = driver.find_element_by_xpath('//*[@id="login-email"]')
element.send_keys('notarealuser@gmail.com')
element = driver.find_element_by_xpath('//*[@id="login-password"]')
element.send_keys('P@$$W0rd123!!')
driver.find_element_by_xpath('//*[@id="login-submit"]').click()


Header

Confirm login success
>>> html = driver.page_source
>>> if "My Network" in html:
...     LoggedIn = True
... else:
...     pass
... 
>>> if LoggedIn:
...     print "Login Successful"
... 
Login Successful