Marcus Lynn

← All posts

Post 14

Deobfuscating Data Dump

(Note: This story takes place in late 2019 -> early 2020. Box’s systems may have changed since then.)

During a legal proceeding there may come a time when a discovery occurs. In layman’s terms and for the purposes of this story its when one must submit all related evidence, files, communications, etc to the other party to ensure a fair trial. For example in a financial case someone might be required to submit their financial records for analysis. While a request for discovery may be objected, it can be enforced by the court and I get the impression that trying to be tricky with the court system is not in your best interest.

They still wanted to be difficult though, and they did it in a clever way. What do you do when you have a combined 75,000 files across various systems and have a court order to deliver them all, while at the same time trying to make hard for your opposition to analyze them? Try this:

  1. Take all your files and combine them into a single folder.
  2. Rename each file in a sequential format, so instead of “Work order XYZ” its “File_000001” then “File_000002”, and so on.
  3. Upload all files to Box.com, an alternative to Dropbox, Google Drive, etc.
  4. Share the access link with the opposing party.
  5. Sit back and profit while the opposing paralegals and consultants cry.

We discovered an interesting behavior with Box.com that isn’t shared with any of it’s competitors: you cannot bulk download files which would total more than 15 gigabytes in size. This means you can’t select all documents, click download, and have all the files. You are forced to scroll through pages of 20 listings maximum and download parts of the file-set at a time. The best part about Box is that you cannot “select all” like you might select all files in a Dropbox transfer/shared link. You must hold the control key and individually click each of the files, then click download to get a .zip of that page’s listings. Approximately 75000 files/20 per page = 3,750 pages of files that one must download to get all the shared files.

This only happens if you share more than 15gb worth of files in a single transfer. The opposition could have done multiple transfers, they could have compressed them into a single file for transport, many things could have been done but they chose not to :)

After timing the consultant who brought this issue to my attention I figured in the average case situation you could operate at 15-20 seconds per page. Factor in human fatigue and it’s almost 16-20 hours of straight effort just trying to get the files downloaded, not including analysis of them or trying to sort out the file names.

To combat this I did two things:

  1. Created a script to automate the download process.
  2. Created another script to rename the files based on the first phrase found in the PDFs after an OCR.

Unfortunately I couldn’t find script #2, but I do have #1- I’ve commented the code and removed any identifying information. Because Box appends a “page=1” or a “page=572” to the end we can predictably know how to traverse all the pages in an automated fashion. A human just needs to tell it what page it needs to stop on. Here’s the flow of the script in human terms:

  1. The script starts by telling Chrome to navigate and load “(URL)?page=1”
  2. Hold down the Control key.
  3. We then click on each file listed on the page x20 times.
  4. Release the Control key.
  5. Find and click the download button.
  6. Wait 20 seconds for the files to download.
  7. Set the next page to visit to be “CurrentURL” plus 1.
    • So “(URL)?page=1” becomes “(URL)?page=2”
  8. Repeat the process now that we are on a new page.
  9. Continue until all pages have been visited & downloaded.

Awesome right? I used Selenium’s Chrome webdriver and the ActionChains library to perform the heavy lifting. This situation occurred another two times so I was able to slightly modify the script each time to fit the new case. I think all in all we probably saved nearly a hundred hours worth of human effort with this one simple trick ;)

######################################## Imports
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.webdriver import ActionChains
######################################## Global Vars
currentBoxPage=1 #Current page the bot is operating on.
numberBoxPages=3750
boxFolderURL_pre = "https://companyname.app.box.com/s/stringofcharacters" #
currentURL = "https://companyname.app.box.com/s/stringofcharacters/folder/stringofcharacters" #Shared link URL given to us by the opposition.
driver = webdriver.Chrome()

########################################
#*********** Functions  ****************
########################################
def main():
    global numberBoxPages
    driver.get("https://companyname.app.box.com/s/stringofcharacters") #Load the Box shared link.
    sleep(15) #This is to give time for a human to enter credentials prior to the script executing.
    for i in range(0, numberBoxPages): #Loop through each page, downloading the files on page and then loading the next page.
        downloadPageFiles()
        nextPage()
    
def downloadPageFiles(): #Downloads the current page by holding control and clicking all 20 file listings on page, then clicking the bulk download button.
    print("Downloading Page: " + str(currentBoxPage))
    driver.get(currentURL) #Gets the website page to be downloaded.
    sleep(1)

    #Click on all the files to be downloaded.
    actionChain = ActionChains(driver)
    actionChain.key_down(Keys.LEFT_CONTROL).perform()
    for i in range(0,20):
        driver.find_element_by_css_selector("div[data-item-index='" + str(i) + "']").click()
        driver.execute_script("window.scrollTo(0, " + str(40*i) + ");")
    actionChain.key_up(Keys.LEFT_CONTROL).perform()

    #Click the "Bulk Download" button.
    actionChain = ActionChains(driver)
    actionChain.move_to_element(driver.find_element_by_css_selector("button[data-resin-target='bulkdownload']"))
    actionChain.click(driver.find_element_by_css_selector("button[data-resin-target='bulkdownload']"))
    actionChain.perform()
    sleep(20) #Wait for the download to complete. Can't go too fast or else Chrome gets scared that you are downloading so many things too quickly.

def nextPage(): #This updates the "page=256" with "page=257" or whatever the next number is in the sequence.
    # You need to update the suffix, page numbers above, and the global vars for each project.
    global boxFolderURL_pre, currentURL, currentBoxPage, numberBoxPages
    currentBoxPage = currentBoxPage + 1
    suffix = "/folder/stringofcharacters?page=" + str(currentBoxPage)
    currentURL = boxFolderURL_pre + suffix
    
########################################
#******** End of Functions  ************
########################################

main()
print("Done")

~Marcus

← All posts