Trouble uploading a document through multipart/form-data via Oracle's UTL_HTTP

I should have mentioned the doc link for UTL_HTTP.WRITE_RAW in my 12.1.0.2 Oracle database.

At first glance, I don't see anything wrong in your request. Did you try uploading the document through the web portal to check if it isn't a document issue?

Thanks Haris for looking into this. Good point about a web test, since I have seen some other forum posts where a corrupted PDF file can cause issues. I successfully uploaded the PDF via the web interface without a problem. The PDF file is just a simple document with only "Test 2" in the body (attached here). Nothing fancy. I initially thought it was something with my request interacting with the package in my sandbox, however, I do get a useful (I think) error if I attempt to upload a base64 encoded copy. (Which I know, you said was not the right way, but was grasping at straws).

SQL> @esign1_base64.sql
Request body>

--THISxisMYxBoundary
Content-Disposition: form-data; name="payload"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

{ "name": "Test_2.pdf"
}
--THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: base64

JVBERi0xLjUNCiW1tb



l0gL1ByZXYgODIyNjUvWFJlZlN0bSA4MTk4OD4+DQpzdGFydHhyZWYNCjgyODIx
DQolJUVPRg==
--THISxisMYxBoundary--
Response> Status Code: 500
Response> Reason Phrase: Internal Server Error
Response> HTTP Version: HTTP/1.1
Response> Server: nginx
Response> Date: Mon, 14 May 2018 14:56:26 GMT
Response> Content-Type: application/json
Response> Transfer-Encoding: chunked
Response> Connection: close
Response> X-Powered-By: Undertow
Response> Allow: GET, POST, HEAD, PUT, PATCH, DELETE
Response body>
{"technical":"DocumentConversionCommand failed and failed retrieving fallback.","messageKey":"error.internal.default","message":"Unexpected error. We apologize for any inconvenience this may have
caused you, please try again. If the problem persists, please contact our support team.","code":500,"name":"Unhandled Server Error"}

PL/SQL procedure successfully completed.

The above does seem to indicate that it was attempting to upload and then threw a conversion error? As another test, to take anything binary out of the picture I attempted to just upload a simple ascii text file, but still getting a 500. Do you have access to the server logs? Or is there a way I can set a flag to get more debug info?

SQL> @esign_text1.sql
Body Length: 353
URL: https://sandbox.esignlive.com/api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/documents

----THISxisMYxBoundary
Content-Disposition: form-data; name="payload"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit


{ "name": "Test1.txt"
}
----THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test1.txt"
Content-Type: text/plain

This is my sample file content!
----THISxisMYxBoundary--
Response> Status Code: 500
Response> Reason Phrase: Internal Server Error
Response> HTTP Version: HTTP/1.1
Response> Server: nginx
Response> Date: Mon, 14 May 2018 14:57:27 GMT
Response> Content-Type: application/octet-stream
Response> Transfer-Encoding: chunked
Response> Connection: close
Response> X-Powered-By: Undertow
Response> Allow: GET, POST, HEAD, PUT, PATCH, DELETE
Response body>
Response length: 27
500: Unhandled Server Error

PL/SQL procedure successfully completed.

Thanks again!

Hi Andy, The esignlive application doesn't accept base64 encoded documents. It definitely will need to be sent as binaries. Can you try again your original test and share the package id? I will go ahead and look at the logs once I've got a package id from you.

Sounds good. I just submitted this one at 1:14 EST. Package id: 2HmThBe9eZopqozAMY63n6TZ5jY=

SQL> @esign_blob1.sql
Request post>
Body Length: 334
Body Length: 82999
Body Length: 26
URL:
https://sandbox.esignlive.com/api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/document
s

----THISxisMYxBoundary
Content-Disposition: form-data;
name="payload"
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 8bit

{ "name": "Test_2.pdf"
}
----THISxisMYxBoundary
Content-Disposition: form-data; name="file";
filename="Test_2.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding:
binary



----THISxisMYxBoundary--
Response> Status Code: 500
Response> Reason Phrase: Internal Server Error
Response> HTTP Version: HTTP/1.1
Response> Server: nginx
Response> Date: Mon, 14 May 2018 17:14:21 GMT
Response> Content-Type: application/octet-stream
Response> Transfer-Encoding: chunked
Response> Connection: close
Response> X-Powered-By: Undertow
Response> Allow: GET, POST, HEAD, PUT, PATCH, DELETE
Response body>
500: Unhandled Server Error

PL/SQL procedure successfully completed.

Thanks for the information. I did submit a request to our maintenance team to retrieve the logs. However, this may take some time. I will get back to you once I have the logs.

Great. Thanks Haris.

Hi Haris, did you ever have any luck tracking down the server logs? A colleague of mine issued a similar request close to my example on some local servers (and after putting in some log requests and some back and forth) was able to track down a server log file with the following output: Sorry about the xxxx's. I never know what is sensitive and what isn't.

  
[5/15/18 11:59:48:645 CDT] 0000027a SessionLogger I   ESIGNLIVE_SESSION_ID: 2d09c1bc7f99be57xxxxxxxxxxxxxxxf3af4b9d8786f1,	USER_UID: pXXXXXXXXX,	REQUEST_PATH: (POST) /packages/Se3OwA7y9dvCgcnlS_EXOFVrtBc=/documents
[5/15/18 11:59:48:663 CDT] 0000027a DocumentResou I   BEGIN: com.silanis.esl.rest.resource.DocumentResource@a74e1353.uploadDocument ( null, Se3OwxxxxxxxxxxXOFVrtBc=, '.toString > 50 chars' )
[5/15/18 11:59:48:663 CDT] 0000027a MultiPartRequ E   Could not read value from header: name
[5/15/18 11:59:48:664 CDT] 0000027a DocumentResou I   END: com.silanis.esl.rest.resource.DocumentResource@a74e1353.uploadDocument elapsed (ms): 2
[5/15/18 11:59:48:664 CDT] 0000027a AbstractEslEx I   User: pXXXXXXXXXX | ValidationException : null : returning status: Bad Request : key=error.validation.invalidParameters TechnicalMessage : Unexpected Content-Disposition value for parameter 'name'

It appears that even though we're setting the Content-Disposition similar to the example in the Documentation perhaps we aren't setting it in the right place? According to this note we should be setting Content-Disposition in the HTTP Header, and not necessarily in the body. Does that make sense to you? UTL_HTTP is pretty low level and finicky if I'm not putting things in the right place. Would it be possible for you to confirm where Content-Disposition should be set and what if anything related should be in the body? I was never really sure I understood why there was a name="payload" and a name = "file" in the doc example. Thanks again!

Hi Andy, From our logs, it looks like you are not sending the pdf binaries to esignlive. Our application tries to convert what you are sending to a pdf and is failing. Can you confirm you are still attempting to upload with pdf binaries and not base64 encoded documents? Also, the headers that you see in our documentation are all required in order to make a successful call. Similar to below:

POST /api/packages/ HTTP/1.1
Host: sanbox.esignlive.com
Connection: keep-alive
Content-Length: 80357
Accept: application/json
Authorization: Basic your_api_key
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarycywBKPMXcPHApu4C

------WebKitFormBoundarycywBKPMXcPHApu4C
Content-Disposition: form-data; name="file"; filename="doc1.pdf"
Content-Type: application/pdf

%PDF-1.5
%µµµµ
1 0 obj
>>>
endobj....

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="payload"

{"name" : "document1"}
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Would you be able to use a tool like Fiddler to monitor the exact request going to esignlive?

Thanks Haris. I'm definitely not doing the base64 thing, but I'm not sure exactly how the binary data is being sent. In Oracle when I try to show what is being sent I get a hexadecimal representation of the data so it looks like I'm sending you something. I'm happy to install Fiddler and do some more troubleshooting on my end. Will keep you posted. In the meantime, I ran another test using the headers in your example. The main difference being: Accept: application/json Whereas the docs show to use text/html. When I changed it to application/json I now get the following error:

{"messageKey":"http.status.406","technical":"","messageLanguage":null,"packageId
":null,"entity":null,"message":"Not
Acceptable","parameters":{},"code":406,"name":"Not Acceptable"}

Which is sort of refreshing :) , just not sure how to fix it. I would think that it would use the package id from the URL . Thanks again!

I had trouble getting Fiddler to run properly on Linux, but did have luck with Charles. Pretty fun, so thanks for that. From the request contents, it does look like I'm sending out the PDF binary. The only thing that I did differently this time was that I re-ordered my parts to match your order in your last example and I also noticed that Connection was set to "close". So, I explicitly set it to "keep-alive" in my header, but still see the "close" line for some reason. Not sure what that would do, since I thought HTTP/1.1 defaults to keep-alive, but thought I would mention it. Does look like application/json is what your server is expecting, so maybe it is a documentation bug.

POST /api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/documents HTTP/1.1
Host: sandbox.esignlive.com
Authorization: Basic xxxxxxxxxxxxxx==
Accept: application/json
Content-Type: multipart/form-data; boundary="----THISxisMYxBoundary"
Content-Length: 83248
Connection: keep-alive
Connection: close


----THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf
%PDF-1.5
%����
1 0 obj

startxref
82821
%%EOF

----THISxisMYxBoundary
Content-Disposition: form-data; name="payload"

{ "name": "Test_2" }
----THISxisMYxBoundary--

The only other thing I can think of is the way that I'm calculating the Content-Length, which is just me adding up all the bytes that I'm sending out. I did try using chunked but got the same message as the previous example:

{"entity":null,"packageId":null,"messageKey":"http.status.406","technical":"","m
essageLanguage":null,"parameters":{},"message":"Not
Acceptable","code":406,"name":"Not Acceptable"}

Fixed the Connection: close issue. I needed to set the persistent connection support flag. UTL_HTTP.set_persistent_conn_support(TRUE); Still getting http.status.406 though

Hi there, Can you try changing the Accept header to "text/html" instead of "application/json"?

Sure thing!

POST /api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/documents HTTP/1.1
Host: sandbox.esignlive.com
Authorization: Basic BlahBlahBlahBlah==
Accept: text/html
Content-Type: multipart/form-data; boundary="----THISxisMYxBoundary"
Content-Length: 83248
Connection: keep-alive


----THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf
%PDF-1.5
%����
1 0 obj
...
startxref
82821
%%EOF

----THISxisMYxBoundary
Content-Disposition: form-data; name="payload"

{ "name": "Test_2" }
----THISxisMYxBoundary--

Returns:

HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Fri, 18 May 2018 19:24:19 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
X-Powered-By: Undertow
Allow: GET, POST, HEAD, PUT, PATCH, DELETE
Connection: keep-alive

500: Unhandled Server Error

I don't see anything wrong in your request. I will verify again the latest logs and get back to you.

Well, this is sort of embarrassing. (But, in my defense I'm a database guy...) . I tested downloading the PDF again, just to re-confirm that it wasn't a corruption issue and it worked like a charm. I then started to think more about the message that was in the Charles output. "Failed to decode multipart body" . And started to wonder if my boundaries are just dumb. I was using, "----THISxisMYxBoundary" after all. So, I decided to use the one from your example, but noticed that for your multiparts you were injecting an extra "--" ontop of what you were setting as your boundary. After running a simple (non PDF) example, I finally was able upload a file.

POST /api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/documents HTTP/1.1
Host: sandbox.esignlive.com
Authorization: Basic BlahBlahBlah==
Content-Type: multipart/form-data; boundary="----THISxisMYxBoundary"
Content-Length: 357
Connection: keep-alive


------THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test1.txt"
Content-Type: text/plain

This is my sample file content!
------THISxisMYxBoundary
Content-Disposition: form-data; name="payload"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

{ "name": "Test1.txt" }

Finally! However, when I run the same fix for my PDF (the ultimate goal), the multipart error goes away and I get a JSON type message back. Still a 500, but at least something useful this time: Request:

POST /api/packages/2HmThBe9eZopqozAMY63n6TZ5jY=/documents HTTP/1.1
Host: sandbox.esignlive.com
Authorization: Basic BlahBlahBlah==
Accept: text/html
Content-Type: multipart/form-data; boundary="----THISxisMYxBoundary"
Transfer-Encoding: chunked
Connection: keep-alive


------THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf
%PDF-1.5
%����
1 0 obj
...
startxref
82821
%%EOF

------THISxisMYxBoundary
Content-Disposition: form-data; name="payload"

{ "name": "Test_2" }
------THISxisMYxBoundary--

Response:

{"messageKey":"error.internal.default","technical":"65533","message":"Unexpected
error. We apologize for any inconvenience this may have caused you, please try
again. If the problem persists, please contact our support
team.","code":500,"name":"Unhandled Server Error"}

Any idea what the 65533 is about? Maybe this time there might be something useful in the log. Thanks again for hanging in there with me!

Welp. I think I'm good. It appears I was missing a newline after application/pdf: This doesn't work:

------THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf
%PDF-1.5

But, this works like a charm:

------THISxisMYxBoundary
Content-Disposition: form-data; name="file"; filename="Test_2.pdf"
Content-Type: application/pdf

%PDF-1.5
...

Thanks again for all of y our help and patience. I think I'm good for now.

Glad you figured thing out :)