Capture Performance Log using Browermob Proxy

Capture Performance Log and Network Traffic using Browsermob proxy 

BrowserMob Proxy is a free utility to help web developers watch and manipulate network traffic from their AJAX applications. 

It is an open source tool which is allows you to manipulate HTTP requests and responses, capture HTTP content, and export performance data as a HAR file. BMP works well as a standalone proxy server, but it is especially useful when embedded in Selenium tests.

Manually, we can monitor performance of a web application by using browser tools like Firebug or Developers Tools etc. BrowserMob proxy helps us to capture client side performance data for a web application using Selenium WebDriver automated tests. In this example, we will configure BrowserMob proxy for Selenium WebDriver and capture the performance data in HAR format.

HTTP Archive format or HAR is a JSON-formatted archive file format for logging of a web browser's interaction with a web application.

If you're running BrowserMob Proxy within a Java application or Selenium test, get started with Below Example: - 

First download Jar files to Click Here 

// Create Object of BMP and start the proxy
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(7991);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
// start the browser up
WebDriver driver = new ChromeDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "yahoo.com"
proxy.newHar("google.com");
// open yahoo.com
driver.get("http://www.google.com");
// get the HAR data
Har har = proxy.getHar();
// Save Har logs in a File
FileOutputStream fos  = new FileOutputStream("/home/automation/Desktop/google.har");
har.writeTo(fos);

BrowserMob Proxy makes it easy to capture Network logs with Automated Selenium Test , For More Details , Click on BrowserMob GitHub

In above piece of code BrowserMob proxy used below mention methods : -

proxy.start() - Starts the proxy on port 0.

proxy.start(int port) - Starts the proxy on the specified port. Ff the specified proxy has already been started, it throws like mention port already in used.

proxy.enableHarCaptureTypes() - Specify which type of request and response you want to save in from Server .

proxy.stop() - shutdown the proxy server gracefully.

We can use any of the below methods to collect performance data in an HAR format based on the requirement.

Har newHar() - Creates a new HAR file with the default page name.

Har newHar(String initialPageRef) - Creates a new HAR file with the specified 'initialPageRef' as the page name and page title.

Har newHar(String initialPageRef, String initialPageTitle) - Creates a new HAR file with the specified 'initialPageRef' page name and 'initialPageTitle' page title.

Now it's time to run a script for chrome browser and get network logs from server and Save in it a Har file.

Before start this script creation , make sure all dependencies are imported in to the project along with Selenium Server Dependencies .

Click Here to Download : - BrowserMob Proxy 

Click Here to Download : - Selenium Jar 

package chromeBrowser;

import java.io.File;

import java.io.FileOutputStream;

import java.util.List;

import java.util.concurrent.TimeUnit;

import org.apache.commons.io.FileUtils;

import org.openqa.selenium.OutputType;

import org.openqa.selenium.Proxy;

import org.openqa.selenium.TakesScreenshot;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.remote.CapabilityType;

import org.openqa.selenium.remote.DesiredCapabilities;

import net.lightbody.bmp.BrowserMobProxy;

import net.lightbody.bmp.BrowserMobProxyServer;

import net.lightbody.bmp.client.ClientUtil;

import net.lightbody.bmp.core.har.Har;

import net.lightbody.bmp.core.har.HarEntry;

import net.lightbody.bmp.core.har.HarLog;

import net.lightbody.bmp.proxy.CaptureType;

public class ChromeBrowser {

    public static void main(String[] args) throws Exception {

BrowserMobProxy proxy = new BrowserMobProxyServer();

proxy.start(7991);

Proxy seleniumproxy = ClientUtil.createSeleniumProxy(proxy);

proxy.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());

proxy.enableHarCaptureTypes(CaptureType.RESPONSE_HEADERS,

    CaptureType.REQUEST_HEADERS);

DesiredCapabilities caps = new DesiredCapabilities();

caps.setCapability(CapabilityType.PROXY, seleniumproxy);

caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);

caps.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);

proxy.newHar("google.com");

System.setProperty("webdriver.chrome.driver",

    "/home/netstorm/Desktop/chromedriver");

WebDriver driver = new ChromeDriver(caps);

driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);

System.out.println("Browser Opened");

driver.get("https://www.google.com");

Thread.sleep(7000);

System.out.println(driver.getTitle());

Har har = proxy.getHar();

getInfofromHar(har);

FileOutputStream fos = new FileOutputStream(

    "/home/netstorm/Desktop/goole.har");

har.writeTo(fos);

if (proxy != null) {

  proxy.stop();

  driver.quit();

}

}

private static void getInfofromHar(Har har) {

HarLog log = har.getLog();

List<HarEntry> entries = log.getEntries();

for (HarEntry enrty : entries) {

System.out.println("Url name : " + enrty.getRequest().getUrl()

   + "  Urlsize : " + enrty.getRequest().getHeadersSize());

System.out.println("Request method : "

   + enrty.getRequest().getMethod() + " Url status : "

   + enrty.getResponse().getStatus());

System.out.println("Time took : " + enrty.getTime() + " ms");

}

}

}

To analyse Har file open har file --->> Click Here -->> Paste Har log Content >> Click on Preview

Then you will see waterfall model look like this  !!!!!

Test with Headless Browser and BrowserMob Proxy ( PhantomJs) 

We can follow same procedure above mention to import Jar file in IDE  

Selenium Server Dependencies .

Download Browser mob proxy

Download Selenium jar

Download PhantomJS driver, Extract it and Set driver Path just like chrome driver and Other drivers

package HeadlessBrowser;

import java.io.File;

import java.io.FileOutputStream;

import java.util.ArrayList;

import java.util.List;

import java.util.concurrent.TimeUnit;

import org.apache.commons.io.FileUtils;

import org.openqa.selenium.OutputType;

import org.openqa.selenium.TakesScreenshot;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.phantomjs.PhantomJSDriver;

import org.openqa.selenium.phantomjs.PhantomJSDriverService;

import org.openqa.selenium.remote.CapabilityType;

import org.openqa.selenium.remote.DesiredCapabilities;

import net.lightbody.bmp.BrowserMobProxy;

import net.lightbody.bmp.BrowserMobProxyServer;

import net.lightbody.bmp.core.har.Har;

import net.lightbody.bmp.core.har.HarEntry;

import net.lightbody.bmp.core.har.HarLog;

import net.lightbody.bmp.proxy.CaptureType;

public class CaptureHar {

public static void main(String[] args) throws Exception {

  BrowserMobProxy proxy = new BrowserMobProxyServer();

  proxy.start(7991);

  //Proxy seleniumproxy = ClientUtil.createSeleniumProxy(proxy);

  proxy.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());

  proxy.enableHarCaptureTypes(CaptureType.RESPONSE_HEADERS ,

                                            CaptureType.REQUEST_HEADERS);

  ArrayList<String> arl = new ArrayList<>();

  arl.add("--proxy=172.24.2.188:7991");

  arl.add("--ignore-ssl-errors=yes");

  arl.add("--web-security=no");

  DesiredCapabilities caps = new DesiredCapabilities();

// caps.setCapability(CapabilityType.PROXY, seleniumproxy);

  caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);

  caps.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);

  caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, arl);

  caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,

                                           "/home/netstorm/Desktop/phantomjs");

  proxy.newHar("automationeducator.blogspot.com");

  WebDriver driver = new PhantomJSDriver(caps);

  driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);

  System.out.println("Browser Opened");

  driver.get("https://automationeducator.blogspot.com/");

  Thread.sleep(7000);

  System.out.println(driver.getTitle());

  File src = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);

  File dest = new File("/home/automation/Desktop/myblog.png");

  if (dest.exists()) {

   dest.delete();

  }

  FileUtils.moveFile(src, dest);

  Har har = proxy.getHar();

  getInfofromHar(har);

  FileOutputStream fos  =

                      new FileOutputStream("/home/automation/Desktop/myblog.har");

  har.writeTo(fos);

  if ( proxy != null ){

   proxy.stop();

   driver.quit();

  }

}

private static void getInfofromHar(Har har) {

  // TODO Auto-generated method stub

       HarLog log = har.getLog();

       List<HarEntry> entries = log.getEntries();

       for ( HarEntry enrty : entries){     

        System.out.println("Url name : "+enrty.getRequest().getUrl() +

"  Urlsize : "+enrty.getRequest().getHeadersSize());

        System.out.println("Request method : "+enrty.getRequest().getMethod()

  + " Url status : " +enrty.getResponse().getStatus() );

        System.out.println("Time took : " + enrty.getTime() + " ms");

       }

}

}

To analyse Har file open har file --->> Click Here -->> Paste Har log Content >> Click on Preview

Then you will see waterfall model look like this  !!!!!

Test Execute with Firefox Browser and BrowserMob Proxy 

For firefox browser , I am going to use here selenium ( 3.141.59 ) and Browser Mob proxy( 2.1.4 )

So follow same procedure to import jar file in your IDE

Selenium Server Dependencies .

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.FileWriter;
import java.util.List;
import java.util.concurrent.TimeUnit;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.core.har.HarEntry;
import net.lightbody.bmp.core.har.HarLog;
import net.lightbody.bmp.core.har.HarNameValuePair;
import net.lightbody.bmp.proxy.CaptureType;
import org.openqa.selenium.By;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.Proxy.ProxyType;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.remote.CapabilityType;
public class CaptureHarWithFirefox {
public static void main(String[] args) throws Exception {
  System.setProperty("webdriver.gecko.driver", "/home/automation/Back_Up/geckodriver");
  LoggingPreferences logPref = new LoggingPreferences();
  logPref.enable(LogType.PERFORMANCE, Level.ALL);
  BrowserMobProxy proxy = new BrowserMobProxyServer();
  proxy.start(7991);        
  Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
  seleniumProxy.setProxyType(ProxyType.MANUAL);
  seleniumProxy.setHttpProxy("localhost:7991");
  seleniumProxy.setSslProxy("localhost:7991");
  DesiredCapabilities capa = new DesiredCapabilities();
  capa.setCapability(CapabilityType.BROWSER_NAME, "FireFox");
  capa.setCapability(CapabilityType.VERSION, "45");
  capa.setCapability(CapabilityType.PROXY, seleniumProxy);
  WebDriver driver = new FirefoxDriver(capa);
  proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT , CaptureType.REQUEST_HEADERS , CaptureType.RESPONSE_HEADERS);
  driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
  driver.manage().window().maximize();
  proxy.newHar("automationeducator.blogspot.com");
  System.out.println("Browser Opened");
  driver.get("https://automationeducator.blogspot.com/");
  Thread.sleep(7000);
  System.out.println(driver.getTitle());
  File src = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
  File dest = new File("/home/automation/Desktop/myblog.png");
  if (dest.exists()) {
   dest.delete();
  }
  FileUtils.moveFile(src, dest);
  Har har = proxy.getHar();
  getInfofromHar(har);
  FileOutputStream fos  =
                      new FileOutputStream("/home/automation/Desktop/myblog.har");
  har.writeTo(fos);
  if ( proxy != null ){
   proxy.stop();
   driver.quit();
  }
}
private static void getInfofromHar(Har har) {
  // TODO Auto-generated method stub
       HarLog log = har.getLog();
       List<HarEntry> entries = log.getEntries();
       for ( HarEntry enrty : entries){     
        System.out.println("Url name : "+enrty.getRequest().getUrl() +
"  Urlsize : "+enrty.getRequest().getHeadersSize());
        System.out.println("Request method : "+enrty.getRequest().getMethod()
  + " Url status : " +enrty.getResponse().getStatus() );
        System.out.println("Time took : " + enrty.getTime() + " ms");
       }
}
}

To analyse Har file open har file --->> Click Here -->> Paste Har log Content >> Click on Preview

Then you will see waterfall model look like this  !!!!!

Note : - If you are using Selenium 3 version Then some issues with gecko driver to capture Har file to overcome that issues you have to use firefox profile and set into desiredCapability.

Selenium 3 and Firefox do not currently support configuring proxies in an intuitive way. Here's the geckodriver issue: mozilla/geckodriver#97

There are some work-arounds mentioned in the issue. You can also set the proxy on the FirefoxProfile directly (replace 'localhost' with the hostname where the proxy is running.

FirefoxProfile profile = new FirefoxProfile();

profile.setPreference("network.proxy.http", "localhost");

profile.setPreference("network.proxy.http_port", proxy.getPort());

profile.setPreference("network.proxy.ssl", "localhost");

profile.setPreference("network.proxy.ssl_port", proxy.getPort());

profile.setPreference("network.proxy.type", 1);

DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(FirefoxDriver.PROFILE, profile);

driver = new FirefoxDriver(capabilities);

For Example : - 

BrowserMobProxy proxy = new BrowserMobProxyServer();

proxy.start();

// get the Selenium proxy object

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);

// configure it as a desired capability

DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);

FirefoxProfile profile = new FirefoxProfile();

profile.setPreference("network.proxy.http", "localhost");

profile.setPreference("network.proxy.http_port", proxy.getPort());

profile.setPreference("network.proxy.ssl", "localhost");

profile.setPreference("network.proxy.ssl_port", proxy.getPort());

profile.setPreference("network.proxy.type", 1);

// Adding profile

capabilities.setCapability(FirefoxDriver.PROFILE, profile);

// start the browser up

WebDriver driver = new FirefoxDriver(capabilities);

// enable more detailed HAR capture, if desired (see CaptureType for the complete list)

proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);

// create a new HAR with the label "yahoo.com"

proxy.newHar("https://automationeducator.blogspot.com");

// open yahoo.com

driver.get("https://automationeducator.blogspot.com/");

    // get the HAR data

   List<HarEntry> en=proxy.getHar().getLog().getEntries();      

   System.out.println("Size----"+en.size());

int key=1; //Key number

/**Creating the Map Objects for HarEntry to store Key Pair value */

/***Running the loop for HarEntry */

for(HarEntry hr:en){

    System.out.println(hr.getRequest().getUrl());

}

  proxy.stop();

Thanks for visit this blog ....

If you have any query , Please write comment here .