Adding new table with Mozilla's urlClassification -- How do I ensure that table is correctly populated #1091

wesley-tan · 2024-06-23T00:33:40Z

wesley-tan
Jun 23, 2024

I would like to add a table with the corresponding request_id and the firstParty and thirdParty flags under Mozilla's urlClassification
(https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onCompleted)

I am encountering an issue where the firstParty and thirdParty fields are not being populated during my OpenWPM crawl. Instead, these fields contain empty arrays. I have reviewed my crawl setup and logs, but I am unable to identify the root cause of this problem.

I set up OpenWPM environment and ensure all dependencies are installed.
Create a crawl script (crawl.py), and set up browser parameters to run in headless mode.
Observe the resulting SQLite database and notice that the firstParty and thirdParty fields are empty.

I made a new listener:

    // New listener for urlClassification
    this.onUrlClassificationListener = async (details) => {
      // Ignore requests made by extensions
      if (requestStemsFromExtension(details)) {
        return;
      }
      await this.handleUrlClassification(details);
    };
    browser.webRequest.onCompleted.addListener(
      this.onUrlClassificationListener,
      filter,
      ["responseHeaders"],
    );
  }

as well as a new handler

  private async handleUrlClassification(details: browser.webRequest._OnCompletedDetails): Promise<void> {
  console.log('details:', details);

  const urlClassification = details.urlClassification;

  console.log('urlClassification:', urlClassification);

  if (urlClassification) {
    const { firstParty, thirdParty } = urlClassification;

    // Create the classification record
    const classificationRecord: UrlClassification = {
      request_id: Number(details.requestId),
      first_party_classification: JSON.stringify(firstParty),
      third_party_classification: JSON.stringify(thirdParty),
      time_stamp: new Date(details.timeStamp).toISOString(),
    };

    // Save the classification record
    this.dataReceiver.saveRecord("urlclassification", classificationRecord);
  } else {
    // Log if urlClassification is not defined
    console.error('urlClassification is not defined in the details object');
  }
}

but my desired information is still not shown. Clearly, the fields themsevles (e.g. visit_id) are filled up, but the arrays under firstParty and thirdParty always show up as empty. What may be the issue? Thank you/

Answered by wesley-tan

Jun 24, 2024

To whoever faces a similar issue in the future, remember to edit:

schema,sql
parquet_schema.py
http_instrument.ts
test_values.py

View full answer

vringar · 2024-06-23T12:05:01Z

vringar
Jun 23, 2024
Maintainer

console.log('urlClassification:', urlClassification);

Do you see the correct output in the log?

Also you could add something here:

OpenWPM/openwpm/storage/storage_controller.py

Lines 159 to 160 in 566d03b

    
           table_name = TableName(record_type) 
        
           await self.store_record(table_name, visit_id, data)

Like

if table_name == "urlclassification":
    self.logger.error("Got classification record %r", data)

If this still doesn't work, it might make sense to have a look at the generated insert statement

OpenWPM/openwpm/storage/sql_provider.py

Lines 55 to 56 in 566d03b

    
           statement, args = self._generate_insert(table=table, data=record) 
        
           for i in range(len(args)):

0 replies

wesley-tan · 2024-06-23T13:38:06Z

wesley-tan
Jun 23, 2024
Author

Thanks, the urlClassification table seems to be processed as desired

browser_manager      - INFO     - BROWSER 3605402763: EXECUTING COMMAND: InitializeCommand()
browser_manager      - INFO     - BROWSER 3605402763: EXECUTING COMMAND: GetCommand(https://hotjar.com,2)
storage_controller   - ERROR    - Got classification record {'request_id': 18, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.039Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 19, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.095Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 20, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.099Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 21, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.100Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 23, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.101Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 24, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.102Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 22, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.103Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 25, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.110Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 26, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.120Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 28, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.121Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 27, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.138Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 30, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.139Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 29, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.141Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 31, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.143Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 32, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.143Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 33, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.144Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 34, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.146Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 36, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.147Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 35, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.147Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 38, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.148Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 37, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.149Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 62, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.150Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 40, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.151Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 63, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.152Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 64, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.153Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 45, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.154Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 46, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.155Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 47, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.156Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 48, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.157Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 49, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.158Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 50, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.158Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 51, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.159Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 52, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.159Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 53, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.160Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 54, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.161Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 55, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.161Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 57, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.162Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 56, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.162Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 58, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.163Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 59, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.163Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 60, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.164Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 61, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.164Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 44, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.165Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 43, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.166Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 39, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.167Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 42, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.167Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 41, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.168Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 66, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.251Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 65, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.323Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 67, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.542Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 71, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.591Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 78, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.594Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 79, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.595Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 80, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.597Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 81, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.599Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 82, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.599Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 68, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.651Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 70, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.653Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 69, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.658Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 72, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.664Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 84, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.674Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 74, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.681Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 85, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.684Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 86, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.685Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 87, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.686Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 76, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.699Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 73, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.710Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 83, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.724Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 77, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.743Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 75, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.745Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 89, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.797Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 91, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.858Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 88, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.864Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 92, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.869Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 97, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.911Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 94, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.945Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 95, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.952Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 93, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.956Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 96, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:32.981Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 90, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.009Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 101, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.011Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 100, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.017Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 104, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.030Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 102, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.062Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 105, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.094Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 99, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.100Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 106, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.149Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 98, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.226Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 107, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.364Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 108, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.426Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 109, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.435Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 110, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.440Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 115, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.507Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 118, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.508Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 114, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.512Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 116, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.513Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 113, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.529Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 112, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.557Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 111, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.563Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 103, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.565Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 119, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.576Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 120, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.596Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 121, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.614Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 122, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.640Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 124, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.699Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 125, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.723Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 126, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.724Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 128, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.742Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 123, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.759Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 127, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.861Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 129, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.914Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 117, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:33.981Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 131, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.044Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 133, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.079Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 130, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.105Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 136, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.195Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 137, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.206Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 140, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.263Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 139, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.311Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 134, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.317Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 142, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.332Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 143, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.340Z', 'visit_id': 4248551466642229}
CommandSequence for https://admeld.com ran successfully
storage_controller   - ERROR    - Got classification record {'request_id': 132, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.405Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 144, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.417Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 146, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.599Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 138, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.662Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 141, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.674Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 135, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.714Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 149, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.811Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 148, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:34.815Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 145, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.020Z', 'visit_id': 4248551466642229}
browser_manager      - INFO     - BROWSER 3605402763: EXECUTING COMMAND: FinalizeCommand(5)
storage_controller   - ERROR    - Got classification record {'request_id': 147, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.043Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 150, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.087Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 151, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.092Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 153, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.146Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 154, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.147Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 156, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.150Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 152, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.154Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 155, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.160Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 157, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.199Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 161, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.202Z', 'visit_id': 4248551466642229}
storage_controller   - ERROR    - Got classification record {'request_id': 158, 'first_party_classification': '[]', 'third_party_classification': '[]', 'time_stamp': '2024-06-23T13:36:35.390Z', 'visit_id': 4248551466642229}
browser_manager      - INFO     - Finished working on CommandSequence with visit_id 4248551466642229 on browser with id 3605402763
storage_controller   - INFO     - Awaiting all tasks for visit_id 4248551466642229
storage_controller   - INFO     - Terminating handler for Browser-3605402763, because the underlying socket closed
storage_controller   - INFO     - Terminating handler for TaskManager, because the underlying socket closed
storage_controller   - INFO     - Received shutdown signal!
storage_controller   - INFO     - Closing Server
storage_controller   - INFO     - Closed Server
storage_controller   - INFO     - Cancelling status_queue_update
storage_controller   - INFO     - Cancelled status_queue_update
storage_controller   - INFO     - Cancelling timeout_check
storage_controller   - INFO     - Cancelled timeout_check
storage_controller   - INFO     - Starting wait_closed
storage_controller   - INFO     - Completed wait_closed
storage_controller   - INFO     - Entering self.shutdown
CommandSequence for https://hotjar.com ran successfully
storage_controller   - INFO     - structured_storage is shut down

but I'm not sure why firstParty and thirdParty still show up as empty arrays. I wonder if this could be because Enhanced Tracking is disabled, which doesn't allow first Party and thirdParty arrays to be populated.

0 replies

vringar · 2024-06-23T17:36:27Z

vringar
Jun 23, 2024
Maintainer

If you go into the log file that is written to disk, you should also see the output of the Extension.
That should allow you to confirm if the data was successfully collected in the first place and lost during transmission or if the initial collection failed.

I wonder if this could be because Enhanced Tracking is disabled, which doesn't allow first Party and thirdParty arrays to be populated.

I have no particular insight on this setup, but you ofc can't collect data that isn't provided 😁

4 replies

wesley-tan Jun 23, 2024
Author

Thank you for your insight. I will play around with the settings -- I've enabled ETP (including editing configure_firefox.py, etc.) and it still shows up as empty arrays.

wesley-tan Jun 23, 2024
Author

Do you mean the files in datadir? Or way may I find the log file written to disk? Thank you

If you go into the log file that is written to disk, you should also see the output of the Extension.
That should allow you to confirm if the data was successfully collected in the first place and lost during transmission or if the initial collection failed.

vringar Jun 23, 2024
Maintainer

I mean the log path in the TaskManager
https://github.com/openwpm/OpenWPM/blob/master/demo.py#L63
(Please excuse the terse answer, I'm on mobile)
I also remember that this was broken for multiple releases and i don't recall if it's fixed in the latest release or on master

wesley-tan Jun 24, 2024
Author

Thank you so much -- managed to resolve the issue

wesley-tan · 2024-06-24T16:36:03Z

wesley-tan
Jun 24, 2024
Author

To whoever faces a similar issue in the future, remember to edit:

schema,sql
parquet_schema.py
http_instrument.ts
test_values.py

3 replies

vringar Jun 24, 2024
Maintainer

I know this is a bit out there but could you write up your experience with creating a new instrument either as a blog post that the docs could link to or as an .rst in the documentation folder?

wesley-tan Jun 25, 2024
Author

@vringar I'm not too familiar with .rst files but I will write a blog post and update

ndanner-wesleyancs Jul 1, 2024

I can show you the basics of RST.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding new table with Mozilla's urlClassification -- How do I ensure that table is correctly populated #1091

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Adding new table with Mozilla's urlClassification -- How do I ensure that table is correctly populated #1091

wesley-tan Jun 23, 2024

Replies: 4 comments · 7 replies

vringar Jun 23, 2024 Maintainer

wesley-tan Jun 23, 2024 Author

vringar Jun 23, 2024 Maintainer

wesley-tan Jun 23, 2024 Author

wesley-tan Jun 23, 2024 Author

vringar Jun 23, 2024 Maintainer

wesley-tan Jun 24, 2024 Author

wesley-tan Jun 24, 2024 Author

vringar Jun 24, 2024 Maintainer

wesley-tan Jun 25, 2024 Author

ndanner-wesleyancs Jul 1, 2024

wesley-tan
Jun 23, 2024

Replies: 4 comments 7 replies

vringar
Jun 23, 2024
Maintainer

wesley-tan
Jun 23, 2024
Author

vringar
Jun 23, 2024
Maintainer

wesley-tan Jun 23, 2024
Author

wesley-tan Jun 23, 2024
Author

vringar Jun 23, 2024
Maintainer

wesley-tan Jun 24, 2024
Author

wesley-tan
Jun 24, 2024
Author

vringar Jun 24, 2024
Maintainer

wesley-tan Jun 25, 2024
Author