# Populate Elasticsearch Index

This documentation is based on the *FoodCrunch* use case. Please open the link below alongside this page to understand the examples.

{% content-ref url="../../introduction-and-use-case" %}
[introduction-and-use-case](https://docs.squidex.io/id-02-documentation/introduction-and-use-case)
{% endcontent-ref %}

If you are new to *Rules*, we highly recommend reading the following document before continuing:

{% content-ref url="" %}
[](https://docs.squidex.io/id-02-documentation/concepts/rules)
{% endcontent-ref %}

The steps provided here are applicable to both self-hosted installations and Squidex Cloud.

## Prerequisites

In order to create this rule you will need a working installation of Elasticsearch. Elasticsearch can be installed either as a Docker container, or in Kubernetes. Refer to the following links for installation instructions:

* Docker installation: <https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html>
* Kubernetes Installation: <https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-installing-eck.html>

In this example, we are using the cloud version of ElasticSearch at <https://elastic.co>. You can subscribe to a 14 day trial of the Elasticsearch cloud [here](https://www.elastic.co/cloud/elasticsearch-service/signup).

{% hint style="info" %}
Squidex also provides an action for **OpenSearch**. The reason OpenSearch is also available is because Elasticsearch blocks all non-official drivers.&#x20;

OpenSearch action has the same logic and works similarly.
{% endhint %}

### Retrieving ElasticSearch Values for Squidex Rule

1. Select your Elastic deployment from the dashboard.&#x20;

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FnjTOr5j0tVuGQOmOh8aA%2F2023-06-26_19-08.png?alt=media&#x26;token=08839d8b-045c-4ef2-98e7-9e6c0b8ebf64" alt=""><figcaption><p>Select Elastic cluster</p></figcaption></figure></div>

2. Copy the **endpoint** value next to Kibana (2). This is the *Server Url* value which will be required later.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2F9L00xA4qMKbaZEg74YyU%2F2023-06-26_19-22.png?alt=media&#x26;token=61f7f68a-aba5-4b5a-ba38-f51106353bf3" alt=""><figcaption><p>Copy Kibana endpoint</p></figcaption></figure></div>

3. The default user is **elastic**. The password is the same that has been set during Elastic cluster deployment. Make a note of them, these will also be required later in the steps.\
   \
   In case there is a need to reset the password, it can be done from security settings of the deployment. \
   Select **Security** from the deployment and then click **Reset password**.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FONEDLdxqrdELmtugqjoh%2F2023-06-26_19-26.png?alt=media&#x26;token=96bd499c-352b-4ae5-8dc8-dbe02d731c7d" alt=""><figcaption><p>Reset elastic user password (if needed)</p></figcaption></figure></div>

## Creating the Elasticsearch Rule

### Add a Rule

Navigate to **Rules** (1) in the Squidex App and select **+ New Rule** (2).

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FC1ryxhTuGuqxpMIA4hmq%2F2023-05-16_18-19.png?alt=media&#x26;token=ee4524f1-2cc0-45bf-908c-390488862001" alt=""><figcaption><p>Create a new rule</p></figcaption></figure></div>

Select **Content changed** (3) as the trigger for this event. This will load additional settings for the trigger that can be configured such as *Conditions*.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FTl922FWS9eBuGE9oZErV%2F2023-05-16_18-27.png?alt=media&#x26;token=48887ac3-53c1-403c-964a-e19161468091" alt=""><figcaption><p>Select when to trigger the rule</p></figcaption></figure></div>

In this example, we are setting this rule for the `magazine` schema only and triggering it for all events in the schema where the event status is *Published*. Click **+** (4) to add a condition, select the **schema** (5) from the dropdown (i.e. `magazine` in this case) and use the following **condition** (6) in Javascript format:

```
event.status == 'Published'
```

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2F4i1hZ9tkdDxV7vPBBZoK%2F2023-05-16_18-42.png?alt=media&#x26;token=c98523e5-6fb7-40e3-a7cb-2a5700f0b54a" alt=""><figcaption><p>Set the condition for the trigger</p></figcaption></figure></div>

Next, select **Elasticsearch** (7) as the action.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2F2Jq8KMiDSZjsRydqfo5h%2F2023-05-16_18-45.png?alt=media&#x26;token=230df938-dde5-416e-9ed6-5e80ef31669b" alt=""><figcaption><p>Select ElasticSearch as the connector</p></figcaption></figure></div>

Provide the ElasticSearch instance values here such as the Server Url, Index Name, Username, Password etc. Click Save when done.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2F8ZteDTPOkJEGeU2ZLv7v%2F2023-05-16_20-08.png?alt=media&#x26;token=c906886a-ec74-461d-91ec-baaa95018862" alt=""><figcaption><p>Configure ElasticSearch in the rule</p></figcaption></figure></div>

### Test the Rule

In order to test the rule we can either create a new article or update an existing article. The article / content must be in *Published* status for the event to trigger.

On successful trigger, you will see a status message similar to the screenshot below.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FOox0oKMM0Vg8wG0SVxzF%2F2023-05-17_19-00.png?alt=media&#x26;token=64da04fc-7cc2-4414-9046-98af40af0758" alt=""><figcaption><p>Verify that the rule ran</p></figcaption></figure></div>

### Verifying the Indexed Data

On successful *Rule* execution, one can connect to (or query) the ElasticSearch cluster to validate the creation of the documents in the Index. In this example we are using the managed cluster option through <https://cloud.eastic.co>, hence the screenshots and the navigation instructions refer their platform. If you are using a self-hosted ElasticSearch cluster the screenshots may be different.&#x20;

To view the documents, navigate and login to the **Kibana endpoint** (1) (you can find the same in your Elastic Cloud home page).&#x20;

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FXqVqIp01JepTPEUloWB8%2F2023-06-19_20-05.png?alt=media&#x26;token=24c4255c-02e0-41b2-95a2-22f8e3989ba9" alt=""><figcaption><p>Copy Kibana endpoint</p></figcaption></figure></div>

Post login, select **Enterprise Search** (2).

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2Fdoy29B1iPD5n7STyZUAr%2F2023-06-19_20-14.png?alt=media&#x26;token=132bca42-bd31-477c-b0fa-e98919538d52" alt=""><figcaption><p>Access Kibana Dashboard</p></figcaption></figure></div>

Next, click **Indices** (3) and then select the **Index** (4), which in this example is **search-squidex**.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FiH8mxnGurF4sab3PRVKi%2F2023-06-19_20-09.png?alt=media&#x26;token=5606a57c-8d33-4f7e-8be4-63dee8648af6" alt=""><figcaption><p>Select the index</p></figcaption></figure></div>

Select the **Documents** (5) tab to view the documents created through the Rule.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FhWzxuP9i40AKXbRfViRa%2F2023-06-19_20-11.png?alt=media&#x26;token=0bb24bd4-23e4-488f-b33c-04777f57fb98" alt=""><figcaption><p>Select documents tab to view documents</p></figcaption></figure></div>

A newly *Published* article appears as **MagazinePublished** and an *Updated* article will appear as **MagazineUpdated**.

## Customizing the Document

You can choose to customize the document that is populated in ElasticSearch. We support plain *Text*, *Script* and *Liquid* syntax.

Below is an example of custom document that sends only the **Title** of the article. It is written in the Liquid syntax. You must create a valid json string, therefore it is good to use the **escape** formatter.

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2F2R1oUvI9sxPkZGmtYmOq%2F2023-07-10_12-52.png?alt=media&#x26;token=fa811a9e-4873-46a5-b397-1cf1783f8fbb" alt=""><figcaption><p>Custom Document</p></figcaption></figure></div>

## Additional Information

### Important Information about the Workflow

The content ID from Squidex is used as the Elasticsearch document ID.&#x20;

<div align="left"><figure><img src="https://2730255609-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LrL94QfeasZ0wF6YAvB%2Fuploads%2FFaxspooi1zBXbH737OYi%2F2023-05-23_17-06.png?alt=media&#x26;token=698906fa-c91b-4929-90d9-5f4ab39bdab7" alt=""><figcaption><p>Screenshot of ElasticSearch showing document ID</p></figcaption></figure></div>

Populating to ElasticSearch is an **asynchronous** operation. Hence, it may take a few seconds before the updated results appear on a search.

### Using ElasticSearch as the Full Text Store

You can configure Squidex to use ElasticSearch as the full text store and search directly using Squidex search API. To do this set the following values:

```
FULLTEXT__TYPE=elastic
FULLTEXT__ELASTIC__CONFIGURATION=[CONNECTION_STRING]
FULLTEXT__ELASTIC__INDEXNAME=[NAME_OF_THE_INDEX]
```

For more details and other full text parameters, refer to the `appsettings.json` file at: <https://github.com/Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L30>
