Ability to decrypt encrypted files loaded to S3

Related products: None

For customers who are required to encrypt their data at rest in S3, they can use PGP encryption and load the files via an S3 ingest rule or via a Rule with encryption enabled. 

 

If there are an errors in loading the file, it is difficult to troubleshoot the CSV file for import because it is encrypted. As a work around, admins need to export an encrypted file and non-encrypted file for testing from their source system. This creates a lot of additional work. 

It would be better if there was an easier way to decrypt a file. To decrypt the file, an admin would need to have the Private Key for decryption or be able to load the encrypted file through the Rules Engine and see the results. This could work ok if the Rules Engine provides all error logs. But if the file is not loading due to an error, its not possible. The best solution would be to get access to the private key somehow. 

Can you share some of the error messages that you would rather see in the error logs?  

I do not think sharing the key is an option here. cc: @nitin_pawar 


Hi @rakesh - the issues is not more error logs. The problem we’re having is opening the file to investigate the potential issue. For example, if the Rules Engine says there’s bad data in the file (ex: a dropdown doesn’t match in Gainsight), we need to be able to decrypt the file to figure out what it says. As admins, we need access to the files to be able to see what potential issues are with the data file.

For ingesting encrypted data via S3, we setup up an export from our source systems (data mart) to S3. These files get dropped automatically every night and then ingested into Gainsight. If the files run into any errors, the only way to investigate is to produce a second export that is not encrypted.

 

So currently, what we’ve setup is two processes: 

  1. Send an encrypted file each night to S3
  2. Send a second unencrypted file to a shared location each night for admins to troubleshoot

 

It seems that currently having two files is the only available solution because we are not able to get access to the public key. From a security perspective, I understand why Gainsight doesn’t want to share the key. However, it does put us at risk because we have to load a second file to a shared location for troubleshooting. Putting additional unencrypted data on a shared drive for admins to easily access is much more unsecure than sharing the key. 

 


Hi @jean.nairon 

Understood the problem here. 

 

When there is an encrypted file sent to Gainsight, we 

  1. Decrypt it
  2. Parse it 

and then use it in Rules or any other areas. But if we are unable to parse it, you are unable to troubleshoot the problem because the file you have access to is encrypted file. Is this understanding correct?


@rakesh, that’s correct. The file is encrypted so we can’t open it to fully see what the issue might be. Would love to be able to control who has access to the decryption keys. 

 


I would agree. It would great to control who has access to the decryption keys so that only the admin who manages the data connections can access. I have seen admins who manage these connections not have access to extra data from source systems so it can be very difficult to troubleshoot. And sometimes the issues can be as easy as file formatting but we can’t check that without first decrypting the file. 


An update here:

I am working with our security and engineering teams to see what we can do to address this problem.


Thank you for the update @rakesh! Please feel free to reach out to me directly if you have any questions.


Hi,
I am trying to export company object data to s3 using “data designer” as "encrypted pgp file".

Below are the issues:

1. When set up “Export S3” in Data designer, "available encrypted key" is generating encrypted file with .csv extension. Why this file is storing in s3 as .csv instead of .pgp

2. How to decrypt the file which is successfully stored as csv extension in s3 using "available encrypted key" ?

3. Also unbale to generate encrypted file using "custom keys". How to do this ? Our goal is generate "pgp encrypted file" using custom keys using Data designer.

 

​​​​​​​Please advise.


Hi,
I am trying to export company object data to s3 using “data designer” as "encrypted pgp file".

Below are the issues:

1. When set up “Export S3” in Data designer, "available encrypted key" is generating encrypted file with .csv extension. Why this file is storing in s3 as .csv instead of .pgp

File extension doesnt matter @lidwin . You can try opening the CSV file in a text opener and you would see encrypted data. 

2. How to decrypt the file which is successfully stored as csv extension in s3 using "available encrypted key" ?

3. Also unbale to generate encrypted file using "custom keys". How to do this ? Our goal is generate "pgp encrypted file" using custom keys using Data designer.

 

Please advise.

During initial tenant creation, for every customer a PGP key set is prepared specifically for the tenant. If you are not aware of this our Support / Techops should be able to help here. 

For exports, you should be able to use the custom key in the UI in DD and Rules.