> ## Documentation Index > Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-opensw-1781706951-9d0138e.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Unstructured integration > Integrate with the Unstructured document loader using LangChain Python. This notebook covers how to use the Unstructured [document loader](/oss/python/integrations/document_loaders) to load files of many types. `Unstructured` currently supports loading of text files, powerpoints, html, pdfs, images, and more. Please see [Unstructured](/oss/python/integrations/providers/unstructured) for more instructions on setting up Unstructured locally, including setting up required system dependencies. ## Overview ### Integration details | Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/document_loaders/file_loaders/unstructured/) | | :------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------- | :---: | :----------: | :--------------------------------------------------------------------------------------------------: | | [`UnstructuredLoader`](https://reference.langchain.com/python/langchain-unstructured/document_loaders/UnstructuredLoader) | [`langchain-unstructured`](https://reference.langchain.com/python/langchain-unstructured) | ✅ | ❌ | ✅ | ### Loader features | Source | Document Lazy Loading | Native Async Support | | :------------------: | :-------------------: | :------------------: | | `UnstructuredLoader` | ✅ | ❌ | ## Setup ### Credentials By default, `langchain-unstructured` installs a smaller footprint that requires offloading of the partitioning logic to the Unstructured API, which requires an API key. If you use the local installation, you do not need an API key. To get your API key, head over to [this site](https://unstructured.io) and get an API key, and then set it in the cell below: ```python theme={null} import getpass import os if "UNSTRUCTURED_API_KEY" not in os.environ: os.environ["UNSTRUCTURED_API_KEY"] = getpass.getpass( "Enter your Unstructured API key: " ) ``` ### Installation #### Normal installation The following packages are required to run the rest of this notebook. ```python theme={null} # Install package, compatible with API partitioning pip install -qU langchain-unstructured unstructured-client unstructured "unstructured[pdf]" python-magic ``` #### Installation for local If you would like to run the partitioning logic locally, you will need to install a combination of system dependencies, as outlined in the [Unstructured documentation here](https://docs.unstructured.io/open-source/installation/full-installation). For example, on Macs you can install the required dependencies with: ```bash theme={null} # base dependencies brew install libmagic poppler tesseract # If parsing xml / html documents: brew install libxml2 libxslt ``` You can install the required `pip` dependencies needed for local with: ```bash theme={null} pip install "langchain-unstructured[local]" ``` ## Initialization The `UnstructuredLoader` allows loading from a variety of different file types. To read all about the `unstructured` package please refer to their [documentation](https://docs.unstructured.io/open-source/introduction/overview)/. In this example, we show loading from both a text file and a PDF file. ```python theme={null} from langchain_unstructured import UnstructuredLoader file_paths = [ "./example_data/layout-parser-paper.pdf", "./example_data/state_of_the_union.txt", ] loader = UnstructuredLoader(file_paths) ``` ## Load ```python theme={null} docs = loader.load() docs[0] ``` ```text theme={null} INFO: pikepdf C++ to Python logger bridge initialized ``` ```text theme={null} Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}, page_content='1 2 0 2') ``` ```python theme={null} print(docs[0].metadata) ``` ```python theme={null} {'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'} ``` ## Lazy load ```python theme={null} pages = [] for doc in loader.lazy_load(): pages.append(doc) pages[0] ``` ```text theme={null} Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}, page_content='1 2 0 2') ``` ## Post processing If you need to post process the `unstructured` elements after extraction, you can pass in a list of `str` -> `str` functions to the `post_processors` kwarg when you instantiate the `UnstructuredLoader`. This applies to other Unstructured loaders as well. Below is an example. ```python theme={null} from langchain_unstructured import UnstructuredLoader from unstructured.cleaners.core import clean_extra_whitespace loader = UnstructuredLoader( "./example_data/layout-parser-paper.pdf", post_processors=[clean_extra_whitespace], ) docs = loader.load() docs[5:10] ``` ```text theme={null} [Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 393.9), (16.34, 560.0), (36.34, 560.0), (36.34, 393.9)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'parent_id': '89565df026a24279aaea20dc08cedbec', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'e9fa370aef7ee5c05744eb7bb7d9981b'}, page_content='2 v 8 4 3 5 1 . 3 0 1 2 : v i X r a'), Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((157.62199999999999, 114.23496279999995), (157.62199999999999, 146.5141628), (457.7358962799999, 146.5141628), (457.7358962799999, 114.23496279999995)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'bde0b230a1aa488e3ce837d33015181b'}, page_content='LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis'), Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((134.809, 168.64029940800003), (134.809, 192.2517444), (480.5464199080001, 192.2517444), (480.5464199080001, 168.64029940800003)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'parent_id': 'bde0b230a1aa488e3ce837d33015181b', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': '54700f902899f0c8c90488fa8d825bce'}, page_content='Zejiang Shen1 ((cid:0)), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain Lee4, Jacob Carlson3, and Weining Li5'), Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((207.23000000000002, 202.57205439999996), (207.23000000000002, 311.8195408), (408.12676, 311.8195408), (408.12676, 202.57205439999996)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'parent_id': 'bde0b230a1aa488e3ce837d33015181b', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'b650f5867bad9bb4e30384282c79bcfe'}, page_content='1 Allen Institute for AI shannons@allenai.org 2 Brown University ruochen zhang@brown.edu 3 Harvard University {melissadell,jacob carlson}@fas.harvard.edu 4 University of Washington bcgl@cs.washington.edu 5 University of Waterloo w422li@uwaterloo.ca'), Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((162.779, 338.45008160000003), (162.779, 566.8455408), (454.0372021523199, 566.8455408), (454.0372021523199, 338.45008160000003)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'links': [{'text': ':// layout - parser . github . io', 'url': 'https://layout-parser.github.io', 'start_index': 1477}], 'page_number': 1, 'parent_id': 'bde0b230a1aa488e3ce837d33015181b', 'filetype': 'application/pdf', 'category': 'NarrativeText', 'element_id': 'cfc957c94fe63c8fd7c7f4bcb56e75a7'}, page_content='Abstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation. However, various factors like loosely organized codebases and sophisticated model configurations complicate the easy reuse of im- portant innovations by a wide audience. Though there have been on-going efforts to improve reusability and simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper introduces LayoutParser, an open-source library for streamlining the usage of DL in DIA research and applica- tions. The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models for layout de- tection, character recognition, and many other document processing tasks. To promote extensibility, LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digiti- zation pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in real-word use cases. The library is publicly available at https://layout-parser.github.io.')] ``` ## Unstructured API If you want to get up and running with smaller packages and get the most up-to-date partitioning you can `pip install unstructured-client` and `pip install langchain-unstructured`. For more information about the `UnstructuredLoader`, refer to the [Unstructured provider page](https://python.langchain.com/v0.1/docs/integrations/document_loaders/unstructured_file/). The loader will process your document using the hosted Unstructured serverless API when you pass in your `api_key` and set `partition_via_api=True`. You can [generate a free Unstructured API key](https://unstructured.io/api-key/). Check out the [Docker image instructions for self-hosting the Unstructured API](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image) if you’d like to self-host the Unstructured API or run it locally. ```python theme={null} from langchain_unstructured import UnstructuredLoader loader = UnstructuredLoader( file_path="example_data/fake.docx", api_key=os.getenv("UNSTRUCTURED_API_KEY"), partition_via_api=True, ) docs = loader.load() docs[0] ``` ```text theme={null} INFO: Preparing to split document for partition. INFO: Given file doesn't have '.pdf' extension, so splitting is not enabled. INFO: Partitioning without split. INFO: Successfully partitioned the document. ``` ```text theme={null} Document(metadata={'source': 'example_data/fake.docx', 'category_depth': 0, 'filename': 'fake.docx', 'languages': ['por', 'cat'], 'filetype': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'category': 'Title', 'element_id': '56d531394823d81787d77a04462ed096'}, page_content='Lorem ipsum dolor sit amet.') ``` You can also batch multiple files through the Unstructured API in a single API using `UnstructuredLoader`. ```python theme={null} loader = UnstructuredLoader( file_path=["example_data/fake.docx", "example_data/fake-email.eml"], api_key=os.getenv("UNSTRUCTURED_API_KEY"), partition_via_api=True, ) docs = loader.load() print(docs[0].metadata["filename"], ": ", docs[0].page_content[:100]) print(docs[-1].metadata["filename"], ": ", docs[-1].page_content[:100]) ``` ```text theme={null} INFO: Preparing to split document for partition. INFO: Given file doesn't have '.pdf' extension, so splitting is not enabled. INFO: Partitioning without split. INFO: Successfully partitioned the document. INFO: Preparing to split document for partition. INFO: Given file doesn't have '.pdf' extension, so splitting is not enabled. INFO: Partitioning without split. INFO: Successfully partitioned the document. ``` ```text theme={null} fake.docx : Lorem ipsum dolor sit amet. fake-email.eml : Violets are blue ``` ### Unstructured SDK client Partitioning with the Unstructured API relies on the [Unstructured SDK Client](https://docs.unstructured.io/api-reference/api-services/accessing-unstructured-api). If you want to customize the client, you will have to pass an `UnstructuredClient` instance to the `UnstructuredLoader`. Below is an example showing how you can customize features of the client such as using your own `requests.Session()`, passing an alternative `server_url`, and customizing the `RetryConfig` object. For more information about customizing the client or what additional parameters the sdk client accepts, refer to the [Unstructured Python SDK](https://docs.unstructured.io/api-reference/api-services/sdk-python) docs and the client section of the [API Parameters](https://docs.unstructured.io/api-reference/api-services/api-parameters) docs. Note that all API Parameters should be passed to the `UnstructuredLoader`.
Warning: The example below may not use the latest version of the UnstructuredClient and there could be breaking changes in future releases. For the latest examples, refer to the Unstructured Python SDK docs.
```python theme={null} import requests from langchain_unstructured import UnstructuredLoader from unstructured_client import UnstructuredClient from unstructured_client.utils import BackoffStrategy, RetryConfig client = UnstructuredClient( api_key_auth=os.getenv( "UNSTRUCTURED_API_KEY" ), # Note: the client API param is "api_key_auth" instead of "api_key" client=requests.Session(), # Define your own requests session server_url="https://api.unstructuredapp.io/general/v0/general", # Define your own api url retry_config=RetryConfig( strategy="backoff", retry_connection_errors=True, backoff=BackoffStrategy( initial_interval=500, max_interval=60000, exponent=1.5, max_elapsed_time=900000, ), ), # Define your own retry config ) loader = UnstructuredLoader( "./example_data/layout-parser-paper.pdf", partition_via_api=True, client=client, split_pdf_page=True, split_pdf_page_range=[1, 10], ) docs = loader.load() print(docs[0].metadata["filename"], ": ", docs[0].page_content[:100]) ``` ```text theme={null} INFO: Preparing to split document for partition. INFO: Concurrency level set to 5 INFO: Splitting pages 1 to 10 (10 total) INFO: Determined optimal split size of 2 pages. INFO: Partitioning 5 files with 2 page(s) each. INFO: Partitioning set #1 (pages 1-2). INFO: Partitioning set #2 (pages 3-4). INFO: Partitioning set #3 (pages 5-6). INFO: Partitioning set #4 (pages 7-8). INFO: Partitioning set #5 (pages 9-10). INFO: HTTP Request: POST https://api.unstructuredapp.io/general/v0/general "HTTP/1.1 200 OK" INFO: HTTP Request: POST https://api.unstructuredapp.io/general/v0/general "HTTP/1.1 200 OK" INFO: HTTP Request: POST https://api.unstructuredapp.io/general/v0/general "HTTP/1.1 200 OK" INFO: HTTP Request: POST https://api.unstructuredapp.io/general/v0/general "HTTP/1.1 200 OK" INFO: Successfully partitioned set #1, elements added to the final result. INFO: Successfully partitioned set #2, elements added to the final result. INFO: Successfully partitioned set #3, elements added to the final result. INFO: Successfully partitioned set #4, elements added to the final result. INFO: Successfully partitioned set #5, elements added to the final result. INFO: Successfully partitioned the document. ``` ```text theme={null} layout-parser-paper.pdf : LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis ``` ## Chunking The `UnstructuredLoader` does not support `mode` as parameter for grouping text like the older loader `UnstructuredFileLoader` and others did. It instead supports "chunking". Chunking in unstructured differs from other chunking mechanisms you may be familiar with that form chunks based on plain-text features--character sequences like "\n\n" or "\n" that might indicate a paragraph boundary or list-item boundary. Instead, all documents are split using specific knowledge about each document format to partition the document into semantic units (document elements) and we only need to resort to text-splitting when a single element exceeds the desired maximum chunk size. In general, chunking combines consecutive elements to form chunks as large as possible without exceeding the maximum chunk size. Chunking produces a sequence of CompositeElement, Table, or TableChunk elements. Each “chunk” is an instance of one of these three types. See this [page](https://docs.unstructured.io/open-source/core-functionality/chunking) for more details about chunking options, but to reproduce the same behavior as `mode="single"`, you can set `chunking_strategy="basic"`, `max_characters=`, and `include_orig_elements=False`. ```python theme={null} from langchain_unstructured import UnstructuredLoader loader = UnstructuredLoader( "./example_data/layout-parser-paper.pdf", chunking_strategy="basic", max_characters=1000000, include_orig_elements=False, ) docs = loader.load() print("Number of LangChain documents:", len(docs)) print("Length of text in the document:", len(docs[0].page_content)) ``` ```text theme={null} Number of LangChain documents: 1 Length of text in the document: 42772 ``` ## Loading web pages `UnstructuredLoader` accepts a `web_url` kwarg when run locally that populates the `url` parameter of the underlying Unstructured [partition](https://docs.unstructured.io/open-source/core-functionality/partitioning). This allows for the parsing of remotely hosted documents, such as HTML web pages. Example usage: ```python theme={null} from langchain_unstructured import UnstructuredLoader loader = UnstructuredLoader(web_url="https://www.example.com") docs = loader.load() for doc in docs: print(f"{doc}\n") ``` ```python theme={null} page_content='Example Domain' metadata={'category_depth': 0, 'languages': ['eng'], 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'Title', 'element_id': 'fdaa78d856f9d143aeeed85bf23f58f8'} page_content='This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.' metadata={'languages': ['eng'], 'parent_id': 'fdaa78d856f9d143aeeed85bf23f58f8', 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'NarrativeText', 'element_id': '3652b8458b0688639f973fe36253c992'} page_content='More information...' metadata={'category_depth': 0, 'link_texts': ['More information...'], 'link_urls': ['https://www.iana.org/domains/example'], 'languages': ['eng'], 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'Title', 'element_id': '793ab98565d6f6d6f3a6d614e3ace2a9'} ``` *** ## API reference For detailed documentation of all `UnstructuredLoader` features and configurations head to the [API reference](https://reference.langchain.com/python/langchain-unstructured/document_loaders/UnstructuredLoader) ***
[Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers. [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/python/integrations/document_loaders/unstructured_file.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).