Warning: include(/include/othersites.inc): failed to open stream: No such file or directory in /home/content/13/11000713/html/imagemaker-webs/discovery/HowTo/Dedupe.php on line 49

Warning: include(): Failed opening '/include/othersites.inc' for inclusion (include_path='.:/usr/local/php5_6/lib/php') in /home/content/13/11000713/html/imagemaker-webs/discovery/HowTo/Dedupe.php on line 49

Warning: include(/include/menu.inc): failed to open stream: No such file or directory in /home/content/13/11000713/html/imagemaker-webs/discovery/HowTo/Dedupe.php on line 50

Warning: include(): Failed opening '/include/menu.inc' for inclusion (include_path='.:/usr/local/php5_6/lib/php') in /home/content/13/11000713/html/imagemaker-webs/discovery/HowTo/Dedupe.php on line 50

How to Find and Remove Duplicate Documents

Discovery Assistant is an eDiscovery application that can identify and remove duplicate documents from electronic stored documents, including documents from PST files, zip files, scanned documents, Microsoft Office documents, and PDF files.

When document sets are produced as part of an eDiscovery request, it is normally required that duplicate documents be identified and removed. The removal of duplicate documents reduces the overall volume of documents produced, thus reducing the amount of data that needs to be reviewed, and reducing overall litigation costs.

Detection of Duplicate Documents:

As documents are loaded into Discovery Assistant, an MD5 hash value is calculated for each file. The MD5 hash value is a calculated 128 bit binary signature that represents the file contents. Duplicate documents are identified by comparing MD5 hash strings. Documents that have identical MD5 hashes are then binary compared to confirm they match.

Deduplication of documents is usually done at the message level. A message is deemed to be a duplicate only if the message contents and the attachments are exactly the same. Deduplication can also be done at the file level to further reduce the data volume.

To remove duplicates from a document set:

  1. Load the electronic files into Discovery Assistant.
  2. Press the ‘Deduplicate’ button.
  3. Convert the remaining unique documents to Metadata + Text, with Images if desired.
  4. Export the unique documents for review.

Identifying Duplicate Documents Across Multiple Projects:

The Discovery Assistant Terabite application supports grouping multiple Discovery Assistant projects into one large document set in order to identify global duplicates. A global duplicate count is also created at this point.

Other ways to reduce eDiscovery data volume:

Discovery Assistant supports a number of other tools to help reduce data volume and identify key documents.

  • De-Nisting
  • Key-Word Searching
  • Near Duplicates Identification
  • Email Thread Identification
  • Metadata Searching and Filtering

To download and try out Discovery Assistant:

To test the de-duplication capabilities of Discovery Assistant, download and install a demo copy of the program from: http://discoveryassistant.com/Download/Downloads.php

Load in a test set of documents, and deduplicate.

To contact Discovery Assistant:

Email: Sales@DiscoveryAssistant.com
Tel: 604 525-2170
Web: discoveryassistant.com

Other Resources:

http://www.edrm.net/resources/glossary/d/de-duplication

http://en.wikipedia.org/wiki/Deduplication

http://technology.findlaw.com/electronic-discovery/electronic-discovery-guide/processing/stages/deduplication.html

http://whereismydata.wordpress.com/2009/05/28/electronic-discovery-what-is-near-deduplication/

Testimonials:

"About a year ago, I was offered some software with the purchase of a copy machine but the eDiscovery component didn't deduplicate and it was too expensive. I mean, way out there! I'm bullish on Discovery Assistant, though, because it's priced right, easy to use and allows me to print less while doing so much more electronically."
Full service digital printing shop
Now featuring eDiscovery services
Honolulu, HI